Xen 4.0.0-rc8 problem: xenconsole: Could not read tty from store: No such file or directory

Xen 4.0.0-rc8 problem: xenconsole: Could not read tty from store: No such file or directory

[Pasi Kärkkäinen]

Hello,

Michael Young noticed this problem, and I can confirm it:

# xm start f12test -c
xenconsole: Could not read tty from store: No such file or directory

-- Pasi

Re: Xen 4.0.0-rc8 problem: xenconsole: Could not read tty from store: No such file or directory

[Keir Fraser]

I'm able to create HVM and PV domains okay. Are you saying that specifically
xend 'managed domain' functionality is broken? Any idea if that is a recent
regression, if so?

 -- Keir

On 26/03/2010 20:35, "Pasi Kärkkäinen" <pasik@iki.fi> wrote:

> Hello,
> 
> Michael Young noticed this problem, and I can confirm it:
> 
> # xm start f12test -c
> xenconsole: Could not read tty from store: No such file or directory
> 
> -- Pasi
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: Xen 4.0.0-rc8 problem: xenconsole: Could not read tty from store: No such file or directory

[Pasi Kärkkäinen]

On Sat, Mar 27, 2010 at 07:56:00AM +0000, Keir Fraser wrote:
> I'm able to create HVM and PV domains okay. Are you saying that specifically
> xend 'managed domain' functionality is broken? Any idea if that is a recent
> regression, if so?
> 

Michael said it also happens with 'xm create -c', but I'll verify soon..

-- Pasi

>  -- Keir
> 
> On 26/03/2010 20:35, "Pasi Kärkkäinen" <pasik@iki.fi> wrote:
> 
> > Hello,
> > 
> > Michael Young noticed this problem, and I can confirm it:
> > 
> > # xm start f12test -c
> > xenconsole: Could not read tty from store: No such file or directory
> > 
> > -- Pasi
> > 
> > 
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
> 
> 

Re: Xen 4.0.0-rc8 problem: xenconsole: Could not read tty from store: No such file or directory

[Keir Fraser]

On 27/03/2010 10:02, "Pasi Kärkkäinen" <pasik@iki.fi> wrote:

> On Sat, Mar 27, 2010 at 07:56:00AM +0000, Keir Fraser wrote:
>> I'm able to create HVM and PV domains okay. Are you saying that specifically
>> xend 'managed domain' functionality is broken? Any idea if that is a recent
>> regression, if so?
>> 
> 
> Michael said it also happens with 'xm create -c', but I'll verify soon..

That's what I tried, and it seemed to work okay.

 -- Keir

Re: Xen 4.0.0-rc8 problem: xenconsole: Could not read tty from store: No such file or directory

[Pasi Kärkkäinen]

On Sat, Mar 27, 2010 at 10:36:39AM +0000, Keir Fraser wrote:
> On 27/03/2010 10:02, "Pasi Kärkkäinen" <pasik@iki.fi> wrote:
> 
> > On Sat, Mar 27, 2010 at 07:56:00AM +0000, Keir Fraser wrote:
> >> I'm able to create HVM and PV domains okay. Are you saying that specifically
> >> xend 'managed domain' functionality is broken? Any idea if that is a recent
> >> regression, if so?
> >> 
> > 
> > Michael said it also happens with 'xm create -c', but I'll verify soon..
> 
> That's what I tried, and it seemed to work okay.
> 

[root@dom0 xen]# xm create -f /etc/xen/f12test-nonmanaged -c
Using config file "/etc/xen/f12test-nonmanaged".
xenconsole: Could not read tty from store: No such file or directory
[root@dom0 xen]# Started domain f12test (id=1)

[root@dom0 xen]# cat /etc/xen/f12test-nonmanaged
name = "f12test"
uuid = "dbe06b39-a570-2b98-a2fe-8f17738fd1d5"
maxmem = 512
memory = 512
vcpus = 2
bootloader = "/usr/bin/pygrub"
on_poweroff = "destroy"
on_reboot = "restart"
on_crash = "restart"
disk = [ "phy:/dev/vg_f12test/f12test,xvda,w" ]
vif = [ "mac=00:16:36:36:25:bf,bridge=virbr0,script=vif-bridge" ]

And obviously I didn't get to see pygrub or select the domU kernel 
because the console connection failed.. domU did start up fine.

dom0 is Fedora 12 with Xen 4.0.0-rc8 (from Michael's rpm).

So what should I try to debug it? 

-- Pasi

Re: Xen 4.0.0-rc8 problem: xenconsole: Could not read tty from store: No such file or directory

[Keir Fraser]

On 27/03/2010 10:53, "Pasi Kärkkäinen" <pasik@iki.fi> wrote:

> And obviously I didn't get to see pygrub or select the domU kernel
> because the console connection failed.. domU did start up fine.
> 
> dom0 is Fedora 12 with Xen 4.0.0-rc8 (from Michael's rpm).
> 
> So what should I try to debug it?

Assuming worked somewhat recently, I'd bisect. If it hasn't worked recently
I'd wonder what's different/special about those RPMs or test setup. I wonder
if it has something to do with using pygrub? Lots of grub2-support patches
went into that recently.

 -- Keir

Re: Xen 4.0.0-rc8 problem: xenconsole: Could not read tty from store: No such file or directory

[Pasi Kärkkäinen]

On Sat, Mar 27, 2010 at 11:18:55AM +0000, Keir Fraser wrote:
> On 27/03/2010 10:53, "Pasi Kärkkäinen" <pasik@iki.fi> wrote:
> 
> > And obviously I didn't get to see pygrub or select the domU kernel
> > because the console connection failed.. domU did start up fine.
> > 
> > dom0 is Fedora 12 with Xen 4.0.0-rc8 (from Michael's rpm).
> > 
> > So what should I try to debug it?
> 
> Assuming worked somewhat recently, I'd bisect. If it hasn't worked recently
> I'd wonder what's different/special about those RPMs or test setup. I wonder
> if it has something to do with using pygrub? Lots of grub2-support patches
> went into that recently.
> 

I just tried 4.0.0-rc4, and it doesn't work there either:

[root@dom0 ~]# xm create -f /etc/xen/f12test-nonmanaged -c
Using config file "/etc/xen/f12test-nonmanaged".
xenconsole: Could not read tty from store: No such file or directory
[root@dom0 ~]# Started domain f12test (id=1)

About pygrub.. Michael reported pvgrub doesn't work either, it crashes.. 
it might be different issue though.

I'll try installing rc8 from source, let's see if that makes difference.

-- Pasi

Re: Xen 4.0.0-rc8 problem: xenconsole: Could not read tty from store: No such file or directory

[Keir Fraser]

On 27/03/2010 11:24, "Pasi Kärkkäinen" <pasik@iki.fi> wrote:

>> Assuming worked somewhat recently, I'd bisect. If it hasn't worked recently
>> I'd wonder what's different/special about those RPMs or test setup. I wonder
>> if it has something to do with using pygrub? Lots of grub2-support patches
>> went into that recently.
>> 
> 
> I just tried 4.0.0-rc4, and it doesn't work there either:
> 
> [root@dom0 ~]# xm create -f /etc/xen/f12test-nonmanaged -c
> Using config file "/etc/xen/f12test-nonmanaged".
> xenconsole: Could not read tty from store: No such file or directory
> [root@dom0 ~]# Started domain f12test (id=1)
> 
> About pygrub.. Michael reported pvgrub doesn't work either, it crashes..
> it might be different issue though.

If the domain actually boot sokay then I suppose pygrub is unlikely to be
the problem. Not sure it's impossible though, and I would at some point try
booting a kernel in dom0 filespace, just to discount pygrub.

 -- Keir

> I'll try installing rc8 from source, let's see if that makes difference.

Re: Xen 4.0.0-rc8 problem: xenconsole: Could not read tty from store: No such file or directory

[Pasi Kärkkäinen]

On Sat, Mar 27, 2010 at 11:29:25AM +0000, Keir Fraser wrote:
> On 27/03/2010 11:24, "Pasi Kärkkäinen" <pasik@iki.fi> wrote:
> 
> >> Assuming worked somewhat recently, I'd bisect. If it hasn't worked recently
> >> I'd wonder what's different/special about those RPMs or test setup. I wonder
> >> if it has something to do with using pygrub? Lots of grub2-support patches
> >> went into that recently.
> >> 
> > 
> > I just tried 4.0.0-rc4, and it doesn't work there either:
> > 
> > [root@dom0 ~]# xm create -f /etc/xen/f12test-nonmanaged -c
> > Using config file "/etc/xen/f12test-nonmanaged".
> > xenconsole: Could not read tty from store: No such file or directory
> > [root@dom0 ~]# Started domain f12test (id=1)
> > 
> > About pygrub.. Michael reported pvgrub doesn't work either, it crashes..
> > it might be different issue though.
> 
> If the domain actually boot sokay then I suppose pygrub is unlikely to be
> the problem. Not sure it's impossible though, and I would at some point try
> booting a kernel in dom0 filespace, just to discount pygrub.
> 

Ok, I copied the kernel+initramfs to dom0, changed the configuration file,
and now "xm create -c" works!

So it looks like a pygrub problem..

[root@dom0 ]# cat /etc/xen/f12test-nonmanaged
name = "f12test"
uuid = "dbe06b39-a570-2b98-a2fe-8f17738fd1d5"
maxmem = 512
memory = 512
vcpus = 2
#bootloader = "/usr/bin/pygrub"
kernel = "/root/testfoo/vmlinuz-2.6.31.12-174.2.22.fc12.x86_64"
ramdisk = "/root/testfoo/initramfs-2.6.31.12-174.2.22.fc12.x86_64.img"
root = "/dev/mapper/vg_f12test-lv_root"
extra = "ro LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=fi console=hvc0 earlyprintk=xen"
on_poweroff = "destroy"
on_reboot = "restart"
on_crash = "restart"
disk = [ "phy:/dev/vg_f12test/f12test,xvda,w" ]
vif = [ "mac=00:16:36:36:25:bf,bridge=virbr0,script=vif-bridge" ]


-- Pasi

Re: Xen 4.0.0-rc8 problem: xenconsole: Could not read tty from store: No such file or directory

[Ian Campbell]

On Sat, 2010-03-27 at 11:47 +0000, Pasi Kärkkäinen wrote: 
> On Sat, Mar 27, 2010 at 11:29:25AM +0000, Keir Fraser wrote:
> > On 27/03/2010 11:24, "Pasi Kärkkäinen" <pasik@iki.fi> wrote:
> > 
> > >> Assuming worked somewhat recently, I'd bisect. If it hasn't worked recently
> > >> I'd wonder what's different/special about those RPMs or test setup. I wonder
> > >> if it has something to do with using pygrub? Lots of grub2-support patches
> > >> went into that recently.
> > >> 
> > > 
> > > I just tried 4.0.0-rc4, and it doesn't work there either:
> > > 
> > > [root@dom0 ~]# xm create -f /etc/xen/f12test-nonmanaged -c
> > > Using config file "/etc/xen/f12test-nonmanaged".
> > > xenconsole: Could not read tty from store: No such file or directory
> > > [root@dom0 ~]# Started domain f12test (id=1)
> > > 
> > > About pygrub.. Michael reported pvgrub doesn't work either, it crashes..
> > > it might be different issue though.
> > 
> > If the domain actually boot sokay then I suppose pygrub is unlikely to be
> > the problem. Not sure it's impossible though, and I would at some point try
> > booting a kernel in dom0 filespace, just to discount pygrub.
> > 
> 
> Ok, I copied the kernel+initramfs to dom0, changed the configuration file,
> and now "xm create -c" works!
> 
> So it looks like a pygrub problem..

pygrub output should be in the xend logs, it includes the command line
so you can copy that and run by hand in domain 0. Also you can run the
individual pygrub modules (e.g. GrubConf.py) on a config file extracted
from a guest to see if it works as expected (see the end of the file for
the syntax).

Do you know if FC12 uses grub1 or grub2?

IIRC the recent patches were all to grub2 specific code paths.

Ian.

Re: Xen 4.0.0-rc8 problem: xenconsole: Could not read tty from store: No such file or directory

[Keir Fraser]

On 27/03/2010 12:05, "Ian Campbell" <Ian.Campbell@eu.citrix.com> wrote:

>> Ok, I copied the kernel+initramfs to dom0, changed the configuration file,
>> and now "xm create -c" works!
>> 
>> So it looks like a pygrub problem..
> 
> pygrub output should be in the xend logs, it includes the command line
> so you can copy that and run by hand in domain 0. Also you can run the
> individual pygrub modules (e.g. GrubConf.py) on a config file extracted
> from a guest to see if it works as expected (see the end of the file for
> the syntax).
> 
> Do you know if FC12 uses grub1 or grub2?
> 
> IIRC the recent patches were all to grub2 specific code paths.

Pygrub must be running pretty well as the domU does end up booted correctly
and with a console (eventually). This is more likely a console-plumbing
issue, and probably on the xend side I would think.

 -- Keir

Re: Xen 4.0.0-rc8 problem: xenconsole: Could not read tty from store: No such file or directory

[Pasi Kärkkäinen]

On Sat, Mar 27, 2010 at 12:09:03PM +0000, Keir Fraser wrote:
> On 27/03/2010 12:05, "Ian Campbell" <Ian.Campbell@eu.citrix.com> wrote:
> 
> >> Ok, I copied the kernel+initramfs to dom0, changed the configuration file,
> >> and now "xm create -c" works!
> >> 
> >> So it looks like a pygrub problem..
> > 
> > pygrub output should be in the xend logs, it includes the command line
> > so you can copy that and run by hand in domain 0. Also you can run the
> > individual pygrub modules (e.g. GrubConf.py) on a config file extracted
> > from a guest to see if it works as expected (see the end of the file for
> > the syntax).
> > 
> > Do you know if FC12 uses grub1 or grub2?
> > 

F12 uses grub1.

> > IIRC the recent patches were all to grub2 specific code paths.
> 
> Pygrub must be running pretty well as the domU does end up booted correctly
> and with a console (eventually). This is more likely a console-plumbing
> issue, and probably on the xend side I would think.
> 

Yeah, pygrub works in the sense that it's able to fetch the kernel+initrd+cmdlineargs 
from domU and domU boots up ok with them.

I guess that's why I haven't noticed this issue earlier..

Only now I tried to do "xm create -c" to get to pygrub menu to select the kernel 
and noticed this problem..

-- Pasi

Re: Xen 4.0.0-rc8 problem: xenconsole: Could not read tty from store: No such file or directory

[Keir Fraser]

On 27/03/2010 12:14, "Pasi Kärkkäinen" <pasik@iki.fi> wrote:

>> Pygrub must be running pretty well as the domU does end up booted correctly
>> and with a console (eventually). This is more likely a console-plumbing
>> issue, and probably on the xend side I would think.
>> 
> 
> Yeah, pygrub works in the sense that it's able to fetch the
> kernel+initrd+cmdlineargs
> from domU and domU boots up ok with them.
> 
> I guess that's why I haven't noticed this issue earlier..
> 
> Only now I tried to do "xm create -c" to get to pygrub menu to select the
> kernel and noticed this problem..

Does it work with the most recent 3.4 release, do you know?

 -- Keir

Re: Xen 4.0.0-rc8 problem: xenconsole: Could not read tty from store: No such file or directory

[Pasi Kärkkäinen]

On Sat, Mar 27, 2010 at 12:18:54PM +0000, Keir Fraser wrote:
> On 27/03/2010 12:14, "Pasi Kärkkäinen" <pasik@iki.fi> wrote:
> 
> >> Pygrub must be running pretty well as the domU does end up booted correctly
> >> and with a console (eventually). This is more likely a console-plumbing
> >> issue, and probably on the xend side I would think.
> >> 
> > 
> > Yeah, pygrub works in the sense that it's able to fetch the
> > kernel+initrd+cmdlineargs
> > from domU and domU boots up ok with them.
> > 
> > I guess that's why I haven't noticed this issue earlier..
> > 
> > Only now I tried to do "xm create -c" to get to pygrub menu to select the
> > kernel and noticed this problem..
> 
> Does it work with the most recent 3.4 release, do you know?
> 

I'll try latest xen-3.4-testing.hg

-- Pasi

Re: Xen 4.0.0-rc8 problem: xenconsole: Could not read tty from store: No such file or directory

[Pasi Kärkkäinen]

On Sat, Mar 27, 2010 at 12:18:54PM +0000, Keir Fraser wrote:
> On 27/03/2010 12:14, "Pasi Kärkkäinen" <pasik@iki.fi> wrote:
> 
> >> Pygrub must be running pretty well as the domU does end up booted correctly
> >> and with a console (eventually). This is more likely a console-plumbing
> >> issue, and probably on the xend side I would think.
> >> 
> > 
> > Yeah, pygrub works in the sense that it's able to fetch the
> > kernel+initrd+cmdlineargs
> > from domU and domU boots up ok with them.
> > 
> > I guess that's why I haven't noticed this issue earlier..
> > 
> > Only now I tried to do "xm create -c" to get to pygrub menu to select the
> > kernel and noticed this problem..
> 
> Does it work with the most recent 3.4 release, do you know?
> 

Latest xen-3.4-testing.hg seems to work, pygrub menu shows up immediately
after "xm create -c".

-- Pasi

Re: Xen 4.0.0-rc8 problem: xenconsole: Could not read tty from store: No such file or directory

[Keir Fraser]

On 27/03/2010 13:08, "Pasi Kärkkäinen" <pasik@iki.fi> wrote:

>>> Only now I tried to do "xm create -c" to get to pygrub menu to select the
>>> kernel and noticed this problem..
>> 
>> Does it work with the most recent 3.4 release, do you know?
>> 
> 
> Latest xen-3.4-testing.hg seems to work, pygrub menu shows up immediately
> after "xm create -c".

I'll have a look next week, although I'm no expert on the python goop.
Release is now planned for week of 5th April, so there's a bit of leeway to
get this fixed for 4.0.0.

 -- Keir

Re: Xen 4.0.0-rc8 problem: xenconsole: Could not read tty from store: No such file or directory

[Pasi Kärkkäinen]

On Sat, Mar 27, 2010 at 01:14:35PM +0000, Keir Fraser wrote:
> On 27/03/2010 13:08, "Pasi Kärkkäinen" <pasik@iki.fi> wrote:
> 
> >>> Only now I tried to do "xm create -c" to get to pygrub menu to select the
> >>> kernel and noticed this problem..
> >> 
> >> Does it work with the most recent 3.4 release, do you know?
> >> 
> > 
> > Latest xen-3.4-testing.hg seems to work, pygrub menu shows up immediately
> > after "xm create -c".
> 
> I'll have a look next week, although I'm no expert on the python goop.
> Release is now planned for week of 5th April, so there's a bit of leeway to
> get this fixed for 4.0.0.
> 

Ok, thanks!

-- Pasi

[FIXED] Xen 4.0.0-rc8 problem: xenconsole: Could not read tty from store: No such file or directory

[Keir Fraser]

On 27/03/2010 13:20, "Pasi Kärkkäinen" <pasik@iki.fi> wrote:

>>> 
>>> Latest xen-3.4-testing.hg seems to work, pygrub menu shows up immediately
>>> after "xm create -c".
>> 
>> I'll have a look next week, although I'm no expert on the python goop.
>> Release is now planned for week of 5th April, so there's a bit of leeway to
>> get this fixed for 4.0.0.
>> 
> 
> Ok, thanks!

Ok, should be fixed by xen-unstable:21078. It's been broken almost since
3.4.0 was branched off.

 -- Keir

Re: [FIXED] Xen 4.0.0-rc8 problem: xenconsole: Could not read tty from store: No such file or directory

[Pasi Kärkkäinen]

On Sat, Mar 27, 2010 at 04:08:18PM +0000, Keir Fraser wrote:
> On 27/03/2010 13:20, "Pasi Kärkkäinen" <pasik@iki.fi> wrote:
> 
> >>> 
> >>> Latest xen-3.4-testing.hg seems to work, pygrub menu shows up immediately
> >>> after "xm create -c".
> >> 
> >> I'll have a look next week, although I'm no expert on the python goop.
> >> Release is now planned for week of 5th April, so there's a bit of leeway to
> >> get this fixed for 4.0.0.
> >> 
> > 
> > Ok, thanks!
> 
> Ok, should be fixed by xen-unstable:21078. It's been broken almost since
> 3.4.0 was branched off.
> 

I just tried with the latest changes (4.0.0-rc9-pre), and the bug is fixed now.
"xm create -c" connects to the console now, and shows pygrub.

Thanks Keir!

-- Pasi

Re: Xen 4.0.0-rc8 problem: xenconsole: Could not read tty from store: No such file or directory

[Keir Fraser]

On 27/03/2010 11:47, "Pasi Kärkkäinen" <pasik@iki.fi> wrote:

>> If the domain actually boot sokay then I suppose pygrub is unlikely to be
>> the problem. Not sure it's impossible though, and I would at some point try
>> booting a kernel in dom0 filespace, just to discount pygrub.
>> 
> 
> Ok, I copied the kernel+initramfs to dom0, changed the configuration file,
> and now "xm create -c" works!
> 
> So it looks like a pygrub problem..

Yes, or xend's bootloader plumbing. I doubt any devs test pygrub as a matter
of course so this could have been broken for a good while.

 -- Keir

Re: Xen 4.0.0-rc8 problem: xenconsole: Could not read tty from store: No such file or directory

[Pasi Kärkkäinen]

On Sat, Mar 27, 2010 at 12:06:03PM +0000, Keir Fraser wrote:
> On 27/03/2010 11:47, "Pasi Kärkkäinen" <pasik@iki.fi> wrote:
> 
> >> If the domain actually boot sokay then I suppose pygrub is unlikely to be
> >> the problem. Not sure it's impossible though, and I would at some point try
> >> booting a kernel in dom0 filespace, just to discount pygrub.
> >> 
> > 
> > Ok, I copied the kernel+initramfs to dom0, changed the configuration file,
> > and now "xm create -c" works!
> > 
> > So it looks like a pygrub problem..
> 
> Yes, or xend's bootloader plumbing. I doubt any devs test pygrub as a matter
> of course so this could have been broken for a good while.
> 

Yeah.. I just tried reverting pygrub and GrubConf.py to revision 20098 (Aug 20 2009)
before many pygrub changes, but it didn't help. Same problem with the console.

So I guess xend then..

-- Pasi

pvgrub boot problems

[M A Young]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1866 bytes --]

On Sat, 27 Mar 2010, Pasi Kärkkäinen wrote:

> About pygrub.. Michael reported pvgrub doesn't work either, it crashes..
> it might be different issue though.

It is a separate issue because xen-unstable:21078 fixes my pygrub problem, 
but pvgrub still crashes.

With pvgrub it gets as far as the boot menu, but crashes quickly when it 
tries to boot the selection with the error below (this is 32-bit 
pvgrub/guest  on 64-bit OS).

 	Michael Young

root (hd0,0)
Error ENOENT when reading the backend path device/vkbd/0/backend
Page fault at linear address 0x1c, eip 0x51c5e, regs 0x3ff5ff7c, sp 
0x3ff5fff0, our_sp 0x3ff5ff50, code 2
Thread: kbdfront
EIP: 51c5e, EFLAGS 10002.
EBX: 746f6f72 ECX: 00000401 EDX: 00000000
ESI: 3ff3fe98 EDI: 00000001 EBP: 3ff5ffb0 EAX: 00000000
DS: e021 ES: e021 orig_eax: ffffffff, eip: 00051c5e
CS: 1e019 EFLAGS: 00010002 esp: 3ff5fff0 ss: 37eb
base is 0x3ff5ffb0 caller is 0x37eb
base is 0x3ff5fff0 caller is 0x31ad

3ff5ffe0: 00 00 00 00 8f 37 00 00 30 00 00 00 40 f0 ff bf
3ff5fff0: 00 00 00 00 ad 31 00 00 94 fe f3 3f 00 00 00 00
3ff60000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3ff60010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

3ff5ffa0: ff ff ff ff 5e 1c 05 00 19 e0 01 00 02 00 01 00
3ff5ffb0: f0 ff f5 3f eb 37 00 00 00 00 00 00 01 00 00 00
3ff5ffc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3ff5ffd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

51c40: e5 8b 45 08 c7 40 1c 00 00 00 00 c7 40 20 00 00
51c50: 00 00 83 60 18 fe 5d c3 55 89 e5 8b 45 08 c7 40
51c60: 1c 00 00 00 00 c7 40 20 00 00 00 00 83 48 18 01
51c70: 5d c3 55 89 e5 53 83 ec 14 a1 e0 b1 94 00 8b 58
Pagetable walk from virt 1c, base a74000:
   L3 = 0000000119b68027 (0xa75000)  [offset = 0]
    L2 = 0000000119b66067 (0xa77000)  [offset = 0]
     L1 = 0000000000000000 [offset = 0]


[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: pvgrub boot problems

[M A Young]

I have captured earlier parts of a similar boot as well. It looks like 
there might be problems initialising vkbd, however I can successfully use 
the keyboard to manipulate the boot menu.

 	Michael Young

Xen Minimal OS!
   start_info: 0xa71000(VA)
     nr_pages: 0x40000
   shared_inf: 0xdf440000(MA)
      pt_base: 0xa74000(VA)
nr_pt_frames: 0x9
     mfn_list: 0x971000(VA)
    mod_start: 0x0(VA)
      mod_len: 0
        flags: 0x0
     cmd_line: (hd0)/boot/grub/grub.conf
   stack:      0x94fe00-0x96fe00
MM: Init
       _text: 0x0(VA)
      _etext: 0x6aa58(VA)
    _erodata: 0x80000(VA)
      _edata: 0x856d4(VA)
stack start: 0x94fe00(VA)
        _end: 0x9703b4(VA)
   start_pfn: a80
     max_pfn: 3fffd
Mapping memory range 0xc00000 - 0x3fffd000
setting 0x0-0x80000 readonly
skipped 0x1000
MM: Initialise page allocator for c7a000(c7a000)-0(3fffd000)
MM: done
Demand map pfns at 3fffe000-bfffe000.
Heap resides at bffff000-fffff000.
Initialising timer interface
Initialising console ... done.
gnttab_table mapped at 0x3fffe000.
Initialising scheduler
Thread "Idle": pointer: 0xbffff030, stack: 0x3ff10000
Initialising xenbus
Thread "xenstore": pointer: 0xbffff4a0, stack: 0x3ff20000
Dummy main: start_info=0x96ff00
Thread "main": pointer: 0xbffff910, stack: 0x3ff30000
Thread "pcifront": pointer: 0xbffffd80, stack: 0x3ff40000
"main" "(hd0)/boot/grub/grub.conf"
pcifront_watches: waiting for backend path to happear device/pci/0/backend
vbd 51713 is hd0
******************* BLKFRONT for device/vbd/51713 **********


backend at /local/domain/0/backend/vbd/1/51713
Failed to read /local/domain/0/backend/vbd/1/51713/feature-flush-cache.
41943040 sectors of 0 bytes
**************************
Thread "kbdfront": pointer: 0xc0000798, stack: 0x3ff50000
******************* FBFRONT for device/vfb/0 **********


******************* KBDFRONT for device/vkbd/0 **********


Failed to read device/vfb/0/backend-id.
Failed to read device/vkbd/0/backend-id.
error writing initialized: ENOENT

Re: pvgrub boot problems

[Jeremy Fitzhardinge]

On 03/27/2010 12:31 PM, M A Young wrote:
> On Sat, 27 Mar 2010, Pasi Kärkkäinen wrote:
>
>> About pygrub.. Michael reported pvgrub doesn't work either, it crashes..
>> it might be different issue though.
>
> It is a separate issue because xen-unstable:21078 fixes my pygrub 
> problem, but pvgrub still crashes.
>
> With pvgrub it gets as far as the boot menu, but crashes quickly when 
> it tries to boot the selection with the error below (this is 32-bit 
> pvgrub/guest  on 64-bit OS).

I see the same thing, though mostly with 64/64/64 xen/dom0/guest 
boots.   AFAIK Samuel has not been able to repro the problem, even when 
using my kernel and pvgrub images.  I sent him a complete image of my 
/boot; with luck that will work...

     J

>
>     Michael Young
>
> root (hd0,0)
> Error ENOENT when reading the backend path device/vkbd/0/backend
> Page fault at linear address 0x1c, eip 0x51c5e, regs 0x3ff5ff7c, sp 
> 0x3ff5fff0, our_sp 0x3ff5ff50, code 2
> Thread: kbdfront
> EIP: 51c5e, EFLAGS 10002.
> EBX: 746f6f72 ECX: 00000401 EDX: 00000000
> ESI: 3ff3fe98 EDI: 00000001 EBP: 3ff5ffb0 EAX: 00000000
> DS: e021 ES: e021 orig_eax: ffffffff, eip: 00051c5e
> CS: 1e019 EFLAGS: 00010002 esp: 3ff5fff0 ss: 37eb
> base is 0x3ff5ffb0 caller is 0x37eb
> base is 0x3ff5fff0 caller is 0x31ad
>
> 3ff5ffe0: 00 00 00 00 8f 37 00 00 30 00 00 00 40 f0 ff bf
> 3ff5fff0: 00 00 00 00 ad 31 00 00 94 fe f3 3f 00 00 00 00
> 3ff60000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 3ff60010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 3ff5ffa0: ff ff ff ff 5e 1c 05 00 19 e0 01 00 02 00 01 00
> 3ff5ffb0: f0 ff f5 3f eb 37 00 00 00 00 00 00 01 00 00 00
> 3ff5ffc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 3ff5ffd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 51c40: e5 8b 45 08 c7 40 1c 00 00 00 00 c7 40 20 00 00
> 51c50: 00 00 83 60 18 fe 5d c3 55 89 e5 8b 45 08 c7 40
> 51c60: 1c 00 00 00 00 c7 40 20 00 00 00 00 83 48 18 01
> 51c70: 5d c3 55 89 e5 53 83 ec 14 a1 e0 b1 94 00 8b 58
> Pagetable walk from virt 1c, base a74000:
>   L3 = 0000000119b68027 (0xa75000)  [offset = 0]
>    L2 = 0000000119b66067 (0xa77000)  [offset = 0]
>     L1 = 0000000000000000 [offset = 0]
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>    

Re: pvgrub boot problems

[M A Young]

On Sat, 27 Mar 2010, Jeremy Fitzhardinge wrote:

> I see the same thing, though mostly with 64/64/64 xen/dom0/guest boots. 
> AFAIK Samuel has not been able to repro the problem, even when using my 
> kernel and pvgrub images.  I sent him a complete image of my /boot; with luck 
> that will work...

Looking at the logs I think the cases might be slightly different, 
depending on the vfb settings in the guest configuration file.
I don't have any vfb setting as I was using it text-only, and I am 
wondering if pvgrub crashes in these circumstances because it doesn't have 
a vkbd and fails when it tries to clean up a non-existent device.

 	Michael Young

Re: pvgrub boot problems

[Jeremy Fitzhardinge]

On 03/27/2010 03:48 PM, M A Young wrote:
> On Sat, 27 Mar 2010, Jeremy Fitzhardinge wrote:
>
>> I see the same thing, though mostly with 64/64/64 xen/dom0/guest 
>> boots. AFAIK Samuel has not been able to repro the problem, even when 
>> using my kernel and pvgrub images.  I sent him a complete image of my 
>> /boot; with luck that will work...
>
> Looking at the logs I think the cases might be slightly different, 
> depending on the vfb settings in the guest configuration file.
> I don't have any vfb setting as I was using it text-only, and I am 
> wondering if pvgrub crashes in these circumstances because it doesn't 
> have a vkbd and fails when it tries to clean up a non-existent device.

I was wondering if it depends on vfb or not as well.  I haven't really 
got it to work either way, but they do have different failures:

Without vfb the crash is in the "kbdfront" thread (looks like a fairly 
straightforward NULL pointer dereference):

sh-4.0# xm create -c f13pv64
Using config file "/etc/xen/f13pv64".
Started domain f13pv64 (id=1)
                              Xen Minimal OS!
   start_info: 0xa99000(VA)
     nr_pages: 0x20000
   shared_inf: 0xbf450000(MA)
      pt_base: 0xa9c000(VA)
nr_pt_frames: 0x9
     mfn_list: 0x999000(VA)
    mod_start: 0x0(VA)
      mod_len: 0
        flags: 0x0
     cmd_line: (hd0,0)/grub/grub.conf
   stack:      0x958980-0x978980
MM: Init
       _text: 0x0(VA)
      _etext: 0x691c4(VA)
    _erodata: 0x82000(VA)
      _edata: 0x8aae0(VA)
   Booting 'Xen 2.6.32'

root (hd0,0)
Error ENOENT when reading the backend path device/vkbd/0/backend
Page fault at linear address 0x0, rip 0x3a28, regs 0xcfff18, sp 0xcfffc8, our_sp 0xcffed0, code 0
Thread: kbdfront
RIP: e030:[<0000000000003a28>]
RSP: e02b:0000000000cfffc8  EFLAGS: 00010006
RAX: 0000000000000000 RBX: 00000000000039da RCX: 000000000008a240
RDX: 0000002020004590 RSI: 000000000008a180 RDI: 000000000008a9a0
RBP: 0000000000cfffe8 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000a9b000 R11: 0000000000071300 R12: 0000000000cdfe28
R13: 0000000000000001 R14: 0000000000000000 R15: 00000002540be400
base is 0xcfffe8 caller is 0x33da

cfffb0: c8 ff cf 00 00 00 00 00 2b e0 00 00 00 00 00 00
cfffc0: f7 39 00 00 00 00 00 00 da 39 00 00 00 00 00 00
cfffd0: 90 45 00 20 20 00 00 00 e3 8b c4 67 0f 48 00 00
cfffe0: e3 a7 b8 13 0d 48 00 00 00 00 00 00 00 00 00 00

cfffd0: 90 45 00 20 20 00 00 00 e3 8b c4 67 0f 48 00 00
cfffe0: e3 a7 b8 13 0d 48 00 00 00 00 00 00 00 00 00 00
cffff0: da 33 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d00000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

3a10: 04 24 01 48 8b 05 46 55 99 00 44 0f b6 68 01 c6
3a20: 40 01 01 49 8b 44 24 08 48 8b 18 49 83 c4 08 49
3a30: 39 c4 74 19 48 8b 78 f8 e8 85 b1 04 00 48 8b 13
3a40: 49 39 dc 74 08 48 89 d8 48 89 d3 eb e7 48 8b 05
Pagetable walk from virt 0, base a9c000:
  L4 = 000000013446d067 (0xa9d000)  [offset = 0]
   L3 = 000000013446c067 (0xa9e000)  [offset = 0]
    L2 = 000000013446b067 (0xa9f000)  [offset = 0]
     L1 = 0000000000000000 [offset = 0]


The rip in this case corresponds to:

(gdb) list *0x0000000000003a28
0x3a28 is in kbd_thread (/home/jeremy/hg/xen/unstable/stubdom/../extras/mini-os/include/mini-os/wait.h:43).
38	static inline void wake_up(struct wait_queue_head *head)
39	{
40	    unsigned long flags;
41	    struct minios_list_head *tmp, *next;
42	    local_irq_save(flags);
43	    minios_list_for_each_safe(tmp, next,&head->thread_list)
44	    {
45	         struct wait_queue *curr;
46	         curr = minios_list_entry(tmp, struct wait_queue, thread_list);
47	         wake(curr->thread);



But with vfb enabled, I get a crash in "main":

sh-4.0# xm create -c f13pv64
Using config file "/etc/xen/f13pv64".
Started domain f13pv64 (id=2)
                              Xen Minimal OS!
   start_info: 0xa99000(VA)
     nr_pages: 0x20000
   shared_inf: 0xbf450000(MA)
      pt_base: 0xa9c000(VA)
nr_pt_frames: 0x9
     mfn_list: 0x999000(VA)
    mod_start: 0x0(VA)
      mod_len: 0
        flags: 0x0
     cmd_line: (hd0,0)/grub/grub.conf
   stack:      0x958980-0x978980
MM: Init
       _text: 0x0(VA)
      _etext: 0x691c4(VA)
    _erodata: 0x82000(VA)
      _edata: 0x8aae0(VA)
stack start: 0x958980(VA)
        _end: 0x998f88(VA)
   start_pfn: aa8
     max_pfn: 20000
Mapping memory range 0xc00000 - 0x20000000
setting 0x0-0x82000 readonly
skipped 0x1000
MM: Initialise page allocator for ba2000(ba2000)-20000000(20000000)
MM: done
Demand map pfns at 20001000-2020001000.
Heap resides at 2020002000-4020002000.
Initialising timer interface
Initialising console ... done.
gnttab_table mapped at 0x20001000.
Initialising scheduler
Thread "Idle": pointer: 0x2020002050, stack: 0xcb0000
Initialising xenbus
Thread "xenstore": pointer: 0x2020002800, stack: 0xcc0000
Dummy main: start_info=0x978a80
Thread "main": pointer: 0x2020002fb0, stack: 0xcd0000
Thread "pcifront": pointer: 0x2020003760, stack: 0xce0000
"main" "(hd0,0)/grub/grub.conf"
pcifront_watches: waiting for backend path to happear device/pci/0/backend
vbd 51712 is hd0
******************* BLKFRONT for device/vbd/51712 **********


backend at /local/domain/0/backend/vbd/2/51712
Failed to read /local/domain/0/backend/vbd/2/51712/feature-flush-cache.
20971520 sectors of 512 bytes
**************************
Thread "kbdfront": pointer: 0x2020004590, stack: 0xcf0000
******************* FBFRONT for device/vfb/0 **********


******************* KBDFRONT for device/vkbd/0 **********


backend at /local/domain/0/backend/vkbd/2/0
/local/domain/0/backend/vkbd/2/0 connected
************************** KBDFRONT
Thread "kbdfront" exited.
backend at /local/domain/0/backend/vfb/2/0
/local/domain/0/backend/vfb/2/0 connected
************************** FBFRONT
Thread "kbdfront close": pointer: 0x2020004590, stack: 0xcf0000
close fb: backend at /local/domain/0/backend/vfb/2/0
close kbd: backend at /local/domain/0/backend/vkbd/2/0
shutdown_kbdfront: error changing state to 5: ENOENT
Thread "kbdfront close" exited.
   Booting 'Xen 2.6.32'

root (hd0,0)
  Filesystem type is ext2fs, partition type 0x83
kernel /xen.gz console=com1,vga
Page fault at linear address 0x100953340, rip 0x4b798, regs 0xcdfa68, sp 0xcdfb18, our_sp 0xcdfa20, code 2
Thread: main
RIP: e030:[<000000000004b798>]
RSP: e02b:0000000000cdfb18  EFLAGS: 00010293
RAX: 0000000000e2e000 RBX: 0000000000cdfb98 RCX: 0000000100953338
RDX: 0000000000000001 RSI: 0000000000953188 RDI: 0000000000000000
RBP: 0000000000cdfb28 R08: 0000000100953338 R09: 0000000000e2c000
R10: 0000000000007ff0 R11: 0000000000200030 R12: 0000000000000003
R13: 0000002020304e18 R14: 0000000000000000 R15: 0000000000001200
base is 0xcdfb28 caller is 0x57ced
base is 0xcdfb88 caller is 0x3067
base is 0xcdfc88 caller is 0x5c1e3
base is 0xcdfcd8 caller is 0x5be3a
base is 0xcdfcf8 caller is 0x3db6
base is 0xcdfd38 caller is 0x402e
base is 0xcdfd58 caller is 0x8341
base is 0xcdfd98 caller is 0xaa2f
base is 0xcdfdd8 caller is 0x108ca
base is 0xcdfe88 caller is 0x10f62
base is 0xcdff48 caller is 0x4343
base is 0xcdff58 caller is 0x4b4ba
base is 0xcdffe8 caller is 0x33da

cdfb00: 18 fb cd 00 00 00 00 00 2b e0 00 00 00 00 00 00
cdfb10: 18 e8 0d 34 01 00 00 00 25 a0 d8 34 01 00 00 00
cdfb20: 98 fb cd 00 00 00 00 00 88 fb cd 00 00 00 00 00
cdfb30: ed 7c 05 00 00 00 00 00 25 a0 d8 34 01 00 00 00

cdfb10: 18 e8 0d 34 01 00 00 00 25 a0 d8 34 01 00 00 00
cdfb20: 98 fb cd 00 00 00 00 00 88 fb cd 00 00 00 00 00
cdfb30: ed 7c 05 00 00 00 00 00 25 a0 d8 34 01 00 00 00
cdfb40: 30 e8 0d 34 01 00 00 00 03 00 00 00 01 00 00 00

4b780: f2 b9 80 31 95 00 48 8b 04 f1 4c 8b 00 4c 89 04
4b790: f1 48 8b 08 48 8b 70 08 48 89 71 08 39 d7 74 49
4b7a0: be 01 00 00 00 41 b8 80 31 95 00 83 ea 01 8d 4a
4b7b0: 0c 48 89 f3 48 d3 e3 4c 8d 0c 18 41 89 51 10 4c
Pagetable walk from virt 100953340, base a9c000:
  L4 = 000000013446d067 (0xa9d000)  [offset = 0]
   L3 = 0000000000000000 (0xfffffffffffff000)  [offset = 4]
Page fault in pagetable walk (access to invalid memory?).


In this case the rip is:

(gdb) list *0x000000000004b798
0x4b798 is in alloc_pages (mm.c:276).
271	    if ( i == FREELIST_SIZE ) goto no_memory;
272	
273	    /* Unlink a chunk. */
274	    alloc_ch = free_head[i];
275	    free_head[i] = alloc_ch->next;
276	    alloc_ch->next->pprev = alloc_ch->pprev;
277	
278	    /* We may have to break the chunk a number of times. */
279	    while ( i != order )
280	    {

The stack backtrace maps to:

0x57ced:
handle_cow
/home/jeremy/hg/xen/unstable/extras/mini-os/arch/x86/traps.c:147
do_page_fault
/home/jeremy/hg/xen/unstable/extras/mini-os/arch/x86/traps.c:202
0x3067:
error_call_handler
??:0
0x5c1e3:
_realloc_r
/home/jeremy/hg/xen/unstable/stubdom/newlib-x86_64/x86_64-xen-elf/newlib/libc/stdlib/../../../../../newlib-1.16.0/newlib/libc/stdlib/mallocr.c:2947
0x5be3a:
realloc
/home/jeremy/hg/xen/unstable/stubdom/newlib-x86_64/x86_64-xen-elf/newlib/libc/stdlib/../../../../../newlib-1.16.0/newlib/libc/stdlib/realloc.c:19
0x3db6:
load_file
/home/jeremy/hg/xen/unstable/stubdom/grub/mini-os.c:165
0x402e:
load_image
/home/jeremy/hg/xen/unstable/stubdom/grub/mini-os.c:187
0x8341:
kernel_func
/home/jeremy/hg/xen/unstable/stubdom/grub/../grub-upstream/stage2/builtins.c:2713
0xaa2f:
run_script
/home/jeremy/hg/xen/unstable/stubdom/grub/../grub-upstream/stage2/cmdline.c:256
0x108ca:
run_menu
/home/jeremy/hg/xen/unstable/stubdom/grub/../grub-upstream/stage2/stage2.c:769
0x10f62:
cmain
/home/jeremy/hg/xen/unstable/stubdom/grub/../grub-upstream/stage2/stage2.c:1121
0x4343:
main
/home/jeremy/hg/xen/unstable/stubdom/grub/mini-os.c:763
0x4b4ba:
call_main
/home/jeremy/hg/xen/unstable/extras/mini-os/main.c:162
0x33da:
thread_starter
gdtoa-hexnan.c:0

Which looks to me like something about the kernel bzImage is upsetting it.

	J

Re: pvgrub boot problems

[M A Young]

On Sat, 27 Mar 2010, Jeremy Fitzhardinge wrote:

> On 03/27/2010 03:48 PM, M A Young wrote:
>> Looking at the logs I think the cases might be slightly different, 
>> depending on the vfb settings in the guest configuration file.
>> I don't have any vfb setting as I was using it text-only, and I am 
>> wondering if pvgrub crashes in these circumstances because it doesn't have 
>> a vkbd and fails when it tries to clean up a non-existent device.
>
> I was wondering if it depends on vfb or not as well.  I haven't really got it 
> to work either way, but they do have different failures:
>
> Without vfb the crash is in the "kbdfront" thread (looks like a fairly 
> straightforward NULL pointer dereference):

Yes my problem is definitely vfb related. If I throw it a
vfb = [ 'vnc=1' ]
then the boot gets further, but fails because it seems pygrub can't cope 
with gzipped kernels (I get the errors
ERROR Invalid kernel: xc_dom_probe_bzimage_kernel: kernel is not a bzImage
xc_dom_bzimageloader.c:350: panic: xc_dom_probe_bzimage_kernel: kernel is 
not a bzImage
which is true but without working gzip support many guests (probably 
including RHEL5) won't work).

So I have three issues with pvgrub
1) without a vfb setting it crashes
2) it doesn't cope with gzipped kernels
3) it doesn't work well with the
disk = [ 'phy:mapper/VolGroup00-austen1,xvda1,w' ]
option. pygrub sees this as (hd0,0) as I think does the booted OS (though 
I can't say I have actually tested this), but pygrub sees this as (hd0). 
This isn't so important as it is easier to work around.

 	Michael Young

Re: pvgrub boot problems

[Samuel Thibault]

M A Young, le Sat 27 Mar 2010 23:34:58 +0000, a écrit :
> 3) it doesn't work well with the
> disk = [ 'phy:mapper/VolGroup00-austen1,xvda1,w' ]
> option. pygrub sees this as (hd0,0) as I think does the booted OS (though 
> I can't say I have actually tested this), but pygrub sees this as (hd0). 

Yes, I had added documentation about this issue that will be far from
easy to fix within pvgrub itself.

Samuel

Re: pvgrub boot problems

[Samuel Thibault]

Jeremy, just to make sure, does the kbd semaphore patch not fix this
case too?

Jeremy Fitzhardinge, le Sat 27 Mar 2010 16:08:01 -0700, a écrit :
> But with vfb enabled, I get a crash in "main":
> 
> sh-4.0# xm create -c f13pv64
> Using config file "/etc/xen/f13pv64".
> Started domain f13pv64 (id=2)
>                              Xen Minimal OS!
>   start_info: 0xa99000(VA)
>     nr_pages: 0x20000
>   shared_inf: 0xbf450000(MA)
>      pt_base: 0xa9c000(VA)
> nr_pt_frames: 0x9
>     mfn_list: 0x999000(VA)
>    mod_start: 0x0(VA)
>      mod_len: 0
>        flags: 0x0
>     cmd_line: (hd0,0)/grub/grub.conf
>   stack:      0x958980-0x978980
> MM: Init
>       _text: 0x0(VA)
>      _etext: 0x691c4(VA)
>    _erodata: 0x82000(VA)
>      _edata: 0x8aae0(VA)
> stack start: 0x958980(VA)
>        _end: 0x998f88(VA)
>   start_pfn: aa8
>     max_pfn: 20000
> Mapping memory range 0xc00000 - 0x20000000
> setting 0x0-0x82000 readonly
> skipped 0x1000
> MM: Initialise page allocator for ba2000(ba2000)-20000000(20000000)
> MM: done
> Demand map pfns at 20001000-2020001000.
> Heap resides at 2020002000-4020002000.
> Initialising timer interface
> Initialising console ... done.
> gnttab_table mapped at 0x20001000.
> Initialising scheduler
> Thread "Idle": pointer: 0x2020002050, stack: 0xcb0000
> Initialising xenbus
> Thread "xenstore": pointer: 0x2020002800, stack: 0xcc0000
> Dummy main: start_info=0x978a80
> Thread "main": pointer: 0x2020002fb0, stack: 0xcd0000
> Thread "pcifront": pointer: 0x2020003760, stack: 0xce0000
> "main" "(hd0,0)/grub/grub.conf"
> pcifront_watches: waiting for backend path to happear device/pci/0/backend
> vbd 51712 is hd0
> ******************* BLKFRONT for device/vbd/51712 **********
> 
> 
> backend at /local/domain/0/backend/vbd/2/51712
> Failed to read /local/domain/0/backend/vbd/2/51712/feature-flush-cache.
> 20971520 sectors of 512 bytes
> **************************
> Thread "kbdfront": pointer: 0x2020004590, stack: 0xcf0000
> ******************* FBFRONT for device/vfb/0 **********
> 
> 
> ******************* KBDFRONT for device/vkbd/0 **********
> 
> 
> backend at /local/domain/0/backend/vkbd/2/0
> /local/domain/0/backend/vkbd/2/0 connected
> ************************** KBDFRONT
> Thread "kbdfront" exited.
> backend at /local/domain/0/backend/vfb/2/0
> /local/domain/0/backend/vfb/2/0 connected
> ************************** FBFRONT
> Thread "kbdfront close": pointer: 0x2020004590, stack: 0xcf0000
> close fb: backend at /local/domain/0/backend/vfb/2/0
> close kbd: backend at /local/domain/0/backend/vkbd/2/0
> shutdown_kbdfront: error changing state to 5: ENOENT
> Thread "kbdfront close" exited.
>   Booting 'Xen 2.6.32'
> 
> root (hd0,0)
>  Filesystem type is ext2fs, partition type 0x83
> kernel /xen.gz console=com1,vga

kernel /xen.gz?  Do you really want to load the Xen hypervisor?
:) Actually this is odd: if I try the same, I get

xc_dom_bzimageloader.c:350: panic: xc_dom_probe_bzimage_kernel: kernel is not a bzImage
ERROR Invalid kernel: elf_xen_note_check: ERROR: Not a Xen-ELF image: No ELF notes or '__xen_guest' section found.

instead. I do not have any issue booting the PV linux kernels, using a
3.4 hypervisor.

Samuel

> Page fault at linear address 0x100953340, rip 0x4b798, regs 0xcdfa68, sp 
> 0xcdfb18, our_sp 0xcdfa20, code 2
> Thread: main
> RIP: e030:[<000000000004b798>]
> RSP: e02b:0000000000cdfb18  EFLAGS: 00010293
> RAX: 0000000000e2e000 RBX: 0000000000cdfb98 RCX: 0000000100953338
> RDX: 0000000000000001 RSI: 0000000000953188 RDI: 0000000000000000
> RBP: 0000000000cdfb28 R08: 0000000100953338 R09: 0000000000e2c000
> R10: 0000000000007ff0 R11: 0000000000200030 R12: 0000000000000003
> R13: 0000002020304e18 R14: 0000000000000000 R15: 0000000000001200
> base is 0xcdfb28 caller is 0x57ced
> base is 0xcdfb88 caller is 0x3067
> base is 0xcdfc88 caller is 0x5c1e3
> base is 0xcdfcd8 caller is 0x5be3a
> base is 0xcdfcf8 caller is 0x3db6
> base is 0xcdfd38 caller is 0x402e
> base is 0xcdfd58 caller is 0x8341
> base is 0xcdfd98 caller is 0xaa2f
> base is 0xcdfdd8 caller is 0x108ca
> base is 0xcdfe88 caller is 0x10f62
> base is 0xcdff48 caller is 0x4343
> base is 0xcdff58 caller is 0x4b4ba
> base is 0xcdffe8 caller is 0x33da
> 
> cdfb00: 18 fb cd 00 00 00 00 00 2b e0 00 00 00 00 00 00
> cdfb10: 18 e8 0d 34 01 00 00 00 25 a0 d8 34 01 00 00 00
> cdfb20: 98 fb cd 00 00 00 00 00 88 fb cd 00 00 00 00 00
> cdfb30: ed 7c 05 00 00 00 00 00 25 a0 d8 34 01 00 00 00
> 
> cdfb10: 18 e8 0d 34 01 00 00 00 25 a0 d8 34 01 00 00 00
> cdfb20: 98 fb cd 00 00 00 00 00 88 fb cd 00 00 00 00 00
> cdfb30: ed 7c 05 00 00 00 00 00 25 a0 d8 34 01 00 00 00
> cdfb40: 30 e8 0d 34 01 00 00 00 03 00 00 00 01 00 00 00
> 
> 4b780: f2 b9 80 31 95 00 48 8b 04 f1 4c 8b 00 4c 89 04
> 4b790: f1 48 8b 08 48 8b 70 08 48 89 71 08 39 d7 74 49
> 4b7a0: be 01 00 00 00 41 b8 80 31 95 00 83 ea 01 8d 4a
> 4b7b0: 0c 48 89 f3 48 d3 e3 4c 8d 0c 18 41 89 51 10 4c
> Pagetable walk from virt 100953340, base a9c000:
>  L4 = 000000013446d067 (0xa9d000)  [offset = 0]
>   L3 = 0000000000000000 (0xfffffffffffff000)  [offset = 4]
> Page fault in pagetable walk (access to invalid memory?).
> 
> 
> In this case the rip is:
> 
> (gdb) list *0x000000000004b798
> 0x4b798 is in alloc_pages (mm.c:276).
> 271	    if ( i == FREELIST_SIZE ) goto no_memory;
> 272	
> 273	    /* Unlink a chunk. */
> 274	    alloc_ch = free_head[i];
> 275	    free_head[i] = alloc_ch->next;
> 276	    alloc_ch->next->pprev = alloc_ch->pprev;
> 277	
> 278	    /* We may have to break the chunk a number of times. */
> 279	    while ( i != order )
> 280	    {
> 
> The stack backtrace maps to:
> 
> 0x57ced:
> handle_cow
> /home/jeremy/hg/xen/unstable/extras/mini-os/arch/x86/traps.c:147
> do_page_fault
> /home/jeremy/hg/xen/unstable/extras/mini-os/arch/x86/traps.c:202
> 0x3067:
> error_call_handler
> ??:0
> 0x5c1e3:
> _realloc_r
> /home/jeremy/hg/xen/unstable/stubdom/newlib-x86_64/x86_64-xen-elf/newlib/libc/stdlib/../../../../../newlib-1.16.0/newlib/libc/stdlib/mallocr.c:2947
> 0x5be3a:
> realloc
> /home/jeremy/hg/xen/unstable/stubdom/newlib-x86_64/x86_64-xen-elf/newlib/libc/stdlib/../../../../../newlib-1.16.0/newlib/libc/stdlib/realloc.c:19
> 0x3db6:
> load_file
> /home/jeremy/hg/xen/unstable/stubdom/grub/mini-os.c:165
> 0x402e:
> load_image
> /home/jeremy/hg/xen/unstable/stubdom/grub/mini-os.c:187
> 0x8341:
> kernel_func
> /home/jeremy/hg/xen/unstable/stubdom/grub/../grub-upstream/stage2/builtins.c:2713
> 0xaa2f:
> run_script
> /home/jeremy/hg/xen/unstable/stubdom/grub/../grub-upstream/stage2/cmdline.c:256
> 0x108ca:
> run_menu
> /home/jeremy/hg/xen/unstable/stubdom/grub/../grub-upstream/stage2/stage2.c:769
> 0x10f62:
> cmain
> /home/jeremy/hg/xen/unstable/stubdom/grub/../grub-upstream/stage2/stage2.c:1121
> 0x4343:
> main
> /home/jeremy/hg/xen/unstable/stubdom/grub/mini-os.c:763
> 0x4b4ba:
> call_main
> /home/jeremy/hg/xen/unstable/extras/mini-os/main.c:162
> 0x33da:
> thread_starter
> gdtoa-hexnan.c:0
> 
> Which looks to me like something about the kernel bzImage is upsetting it.

Re: pvgrub boot problems

[Jeremy Fitzhardinge]

On 04/05/2010 04:57 PM, Samuel Thibault wrote:
> Jeremy, just to make sure, does the kbd semaphore patch not fix this
> case too?
>    

No, afraid not.  I applied all the patches you posted today but I get 
the same failure when using vfb.

But! With vfb disabled I could successfully boot.  So there's some progress.

     J

> Jeremy Fitzhardinge, le Sat 27 Mar 2010 16:08:01 -0700, a écrit :
>    
>> But with vfb enabled, I get a crash in "main":
>>
>> sh-4.0# xm create -c f13pv64
>> Using config file "/etc/xen/f13pv64".
>> Started domain f13pv64 (id=2)
>>                               Xen Minimal OS!
>>    start_info: 0xa99000(VA)
>>      nr_pages: 0x20000
>>    shared_inf: 0xbf450000(MA)
>>       pt_base: 0xa9c000(VA)
>> nr_pt_frames: 0x9
>>      mfn_list: 0x999000(VA)
>>     mod_start: 0x0(VA)
>>       mod_len: 0
>>         flags: 0x0
>>      cmd_line: (hd0,0)/grub/grub.conf
>>    stack:      0x958980-0x978980
>> MM: Init
>>        _text: 0x0(VA)
>>       _etext: 0x691c4(VA)
>>     _erodata: 0x82000(VA)
>>       _edata: 0x8aae0(VA)
>> stack start: 0x958980(VA)
>>         _end: 0x998f88(VA)
>>    start_pfn: aa8
>>      max_pfn: 20000
>> Mapping memory range 0xc00000 - 0x20000000
>> setting 0x0-0x82000 readonly
>> skipped 0x1000
>> MM: Initialise page allocator for ba2000(ba2000)-20000000(20000000)
>> MM: done
>> Demand map pfns at 20001000-2020001000.
>> Heap resides at 2020002000-4020002000.
>> Initialising timer interface
>> Initialising console ... done.
>> gnttab_table mapped at 0x20001000.
>> Initialising scheduler
>> Thread "Idle": pointer: 0x2020002050, stack: 0xcb0000
>> Initialising xenbus
>> Thread "xenstore": pointer: 0x2020002800, stack: 0xcc0000
>> Dummy main: start_info=0x978a80
>> Thread "main": pointer: 0x2020002fb0, stack: 0xcd0000
>> Thread "pcifront": pointer: 0x2020003760, stack: 0xce0000
>> "main" "(hd0,0)/grub/grub.conf"
>> pcifront_watches: waiting for backend path to happear device/pci/0/backend
>> vbd 51712 is hd0
>> ******************* BLKFRONT for device/vbd/51712 **********
>>
>>
>> backend at /local/domain/0/backend/vbd/2/51712
>> Failed to read /local/domain/0/backend/vbd/2/51712/feature-flush-cache.
>> 20971520 sectors of 512 bytes
>> **************************
>> Thread "kbdfront": pointer: 0x2020004590, stack: 0xcf0000
>> ******************* FBFRONT for device/vfb/0 **********
>>
>>
>> ******************* KBDFRONT for device/vkbd/0 **********
>>
>>
>> backend at /local/domain/0/backend/vkbd/2/0
>> /local/domain/0/backend/vkbd/2/0 connected
>> ************************** KBDFRONT
>> Thread "kbdfront" exited.
>> backend at /local/domain/0/backend/vfb/2/0
>> /local/domain/0/backend/vfb/2/0 connected
>> ************************** FBFRONT
>> Thread "kbdfront close": pointer: 0x2020004590, stack: 0xcf0000
>> close fb: backend at /local/domain/0/backend/vfb/2/0
>> close kbd: backend at /local/domain/0/backend/vkbd/2/0
>> shutdown_kbdfront: error changing state to 5: ENOENT
>> Thread "kbdfront close" exited.
>>    Booting 'Xen 2.6.32'
>>
>> root (hd0,0)
>>   Filesystem type is ext2fs, partition type 0x83
>> kernel /xen.gz console=com1,vga
>>      
> kernel /xen.gz?  Do you really want to load the Xen hypervisor?
> :) Actually this is odd: if I try the same, I get
>
> xc_dom_bzimageloader.c:350: panic: xc_dom_probe_bzimage_kernel: kernel is not a bzImage
> ERROR Invalid kernel: elf_xen_note_check: ERROR: Not a Xen-ELF image: No ELF notes or '__xen_guest' section found.
>
> instead. I do not have any issue booting the PV linux kernels, using a
> 3.4 hypervisor.
>
> Samuel
>
>    
>> Page fault at linear address 0x100953340, rip 0x4b798, regs 0xcdfa68, sp
>> 0xcdfb18, our_sp 0xcdfa20, code 2
>> Thread: main
>> RIP: e030:[<000000000004b798>]
>> RSP: e02b:0000000000cdfb18  EFLAGS: 00010293
>> RAX: 0000000000e2e000 RBX: 0000000000cdfb98 RCX: 0000000100953338
>> RDX: 0000000000000001 RSI: 0000000000953188 RDI: 0000000000000000
>> RBP: 0000000000cdfb28 R08: 0000000100953338 R09: 0000000000e2c000
>> R10: 0000000000007ff0 R11: 0000000000200030 R12: 0000000000000003
>> R13: 0000002020304e18 R14: 0000000000000000 R15: 0000000000001200
>> base is 0xcdfb28 caller is 0x57ced
>> base is 0xcdfb88 caller is 0x3067
>> base is 0xcdfc88 caller is 0x5c1e3
>> base is 0xcdfcd8 caller is 0x5be3a
>> base is 0xcdfcf8 caller is 0x3db6
>> base is 0xcdfd38 caller is 0x402e
>> base is 0xcdfd58 caller is 0x8341
>> base is 0xcdfd98 caller is 0xaa2f
>> base is 0xcdfdd8 caller is 0x108ca
>> base is 0xcdfe88 caller is 0x10f62
>> base is 0xcdff48 caller is 0x4343
>> base is 0xcdff58 caller is 0x4b4ba
>> base is 0xcdffe8 caller is 0x33da
>>
>> cdfb00: 18 fb cd 00 00 00 00 00 2b e0 00 00 00 00 00 00
>> cdfb10: 18 e8 0d 34 01 00 00 00 25 a0 d8 34 01 00 00 00
>> cdfb20: 98 fb cd 00 00 00 00 00 88 fb cd 00 00 00 00 00
>> cdfb30: ed 7c 05 00 00 00 00 00 25 a0 d8 34 01 00 00 00
>>
>> cdfb10: 18 e8 0d 34 01 00 00 00 25 a0 d8 34 01 00 00 00
>> cdfb20: 98 fb cd 00 00 00 00 00 88 fb cd 00 00 00 00 00
>> cdfb30: ed 7c 05 00 00 00 00 00 25 a0 d8 34 01 00 00 00
>> cdfb40: 30 e8 0d 34 01 00 00 00 03 00 00 00 01 00 00 00
>>
>> 4b780: f2 b9 80 31 95 00 48 8b 04 f1 4c 8b 00 4c 89 04
>> 4b790: f1 48 8b 08 48 8b 70 08 48 89 71 08 39 d7 74 49
>> 4b7a0: be 01 00 00 00 41 b8 80 31 95 00 83 ea 01 8d 4a
>> 4b7b0: 0c 48 89 f3 48 d3 e3 4c 8d 0c 18 41 89 51 10 4c
>> Pagetable walk from virt 100953340, base a9c000:
>>   L4 = 000000013446d067 (0xa9d000)  [offset = 0]
>>    L3 = 0000000000000000 (0xfffffffffffff000)  [offset = 4]
>> Page fault in pagetable walk (access to invalid memory?).
>>
>>
>> In this case the rip is:
>>
>> (gdb) list *0x000000000004b798
>> 0x4b798 is in alloc_pages (mm.c:276).
>> 271	    if ( i == FREELIST_SIZE ) goto no_memory;
>> 272	
>> 273	    /* Unlink a chunk. */
>> 274	    alloc_ch = free_head[i];
>> 275	    free_head[i] = alloc_ch->next;
>> 276	    alloc_ch->next->pprev = alloc_ch->pprev;
>> 277	
>> 278	    /* We may have to break the chunk a number of times. */
>> 279	    while ( i != order )
>> 280	    {
>>
>> The stack backtrace maps to:
>>
>> 0x57ced:
>> handle_cow
>> /home/jeremy/hg/xen/unstable/extras/mini-os/arch/x86/traps.c:147
>> do_page_fault
>> /home/jeremy/hg/xen/unstable/extras/mini-os/arch/x86/traps.c:202
>> 0x3067:
>> error_call_handler
>> ??:0
>> 0x5c1e3:
>> _realloc_r
>> /home/jeremy/hg/xen/unstable/stubdom/newlib-x86_64/x86_64-xen-elf/newlib/libc/stdlib/../../../../../newlib-1.16.0/newlib/libc/stdlib/mallocr.c:2947
>> 0x5be3a:
>> realloc
>> /home/jeremy/hg/xen/unstable/stubdom/newlib-x86_64/x86_64-xen-elf/newlib/libc/stdlib/../../../../../newlib-1.16.0/newlib/libc/stdlib/realloc.c:19
>> 0x3db6:
>> load_file
>> /home/jeremy/hg/xen/unstable/stubdom/grub/mini-os.c:165
>> 0x402e:
>> load_image
>> /home/jeremy/hg/xen/unstable/stubdom/grub/mini-os.c:187
>> 0x8341:
>> kernel_func
>> /home/jeremy/hg/xen/unstable/stubdom/grub/../grub-upstream/stage2/builtins.c:2713
>> 0xaa2f:
>> run_script
>> /home/jeremy/hg/xen/unstable/stubdom/grub/../grub-upstream/stage2/cmdline.c:256
>> 0x108ca:
>> run_menu
>> /home/jeremy/hg/xen/unstable/stubdom/grub/../grub-upstream/stage2/stage2.c:769
>> 0x10f62:
>> cmain
>> /home/jeremy/hg/xen/unstable/stubdom/grub/../grub-upstream/stage2/stage2.c:1121
>> 0x4343:
>> main
>> /home/jeremy/hg/xen/unstable/stubdom/grub/mini-os.c:763
>> 0x4b4ba:
>> call_main
>> /home/jeremy/hg/xen/unstable/extras/mini-os/main.c:162
>> 0x33da:
>> thread_starter
>> gdtoa-hexnan.c:0
>>
>> Which looks to me like something about the kernel bzImage is upsetting it.
>>      
>    

Re: pvgrub boot problems

[Samuel Thibault]

Jeremy Fitzhardinge, le Mon 05 Apr 2010 17:28:47 -0700, a écrit :
> On 04/05/2010 04:57 PM, Samuel Thibault wrote:
> >Jeremy, just to make sure, does the kbd semaphore patch not fix this
> >case too?
> >   
> 
> No, afraid not.  I applied all the patches you posted today but I get 
> the same failure when using vfb.

Ah.  So there must be an additional bug in vfb, which I have never
triggered since I've never used vfb recently :)

Samuel

Re: pvgrub boot problems

[Samuel Thibault]

Jeremy Fitzhardinge, le Mon 05 Apr 2010 17:28:47 -0700, a écrit :
> On 04/05/2010 04:57 PM, Samuel Thibault wrote:
> >Jeremy, just to make sure, does the kbd semaphore patch not fix this
> >case too?
> >   
> 
> No, afraid not.  I applied all the patches you posted today but I get 
> the same failure when using vfb.

Could you try commenting the two free_page() calls in fbfront.c?

Samuel

Re: pvgrub boot problems

[Samuel Thibault]

Samuel Thibault, le Tue 06 Apr 2010 22:28:55 +0200, a écrit :
> Jeremy Fitzhardinge, le Mon 05 Apr 2010 17:28:47 -0700, a écrit :
> > On 04/05/2010 04:57 PM, Samuel Thibault wrote:
> > >Jeremy, just to make sure, does the kbd semaphore patch not fix this
> > >case too?
> > >   
> > 
> > No, afraid not.  I applied all the patches you posted today but I get 
> > the same failure when using vfb.
> 
> Could you try commenting the two free_page() calls in fbfront.c?

(BTW, this is an example where using grants instead of MFNs in the
interface makes things simpler to make sure who is still writing to a
shared ring...)

Samuel

Re: pvgrub boot problems

[Samuel Thibault]

Jeremy Fitzhardinge, le Sat 27 Mar 2010 15:23:42 -0700, a écrit :
> AFAIK Samuel has not been able to repro the problem

BTW, one thing probably worth noting: I'm still using the Xen 3.2
hypervisor on my debian laptop.  I'll try with 3.4 and 4.0.  I have had
reports of GNU Mach systematically failing on them too.

Samuel

Re: pvgrub boot problems

[Samuel Thibault]

Jeremy Fitzhardinge, le Sat 27 Mar 2010 15:23:42 -0700, a écrit :
> >Page fault at linear address 0x1c, eip 0x51c5e, regs 0x3ff5ff7c, sp 
> >0x3ff5fff0, our_sp 0x3ff5ff50, code 2
> >Thread: kbdfront

This should be fixed by the patch I've just sent.

Samuel

Re: pvgrub boot problems

[M A Young]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 567 bytes --]

On Sun, 4 Apr 2010, Samuel Thibault wrote:

> Jeremy Fitzhardinge, le Sat 27 Mar 2010 15:23:42 -0700, a écrit :
>>> Page fault at linear address 0x1c, eip 0x51c5e, regs 0x3ff5ff7c, sp
>>> 0x3ff5fff0, our_sp 0x3ff5ff50, code 2
>>> Thread: kbdfront
>
> This should be fixed by the patch I've just sent.

Yes, I have applied that and the patch you posted just before it and it 
fixes the crash issue if there is no frame buffer. The boot still fails 
for me because of the gzip issue but it didn't get that far previously 
with this setup.

 	Michael Young

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: pvgrub boot problems

[Samuel Thibault]

M A Young, le Mon 05 Apr 2010 19:05:33 +0100, a écrit :
> Yes, I have applied that and the patch you posted just before it and it 
> fixes the crash issue if there is no frame buffer. The boot still fails 
> for me because of the gzip issue but it didn't get that far previously 
> with this setup.

The gzip issue?  Is your kernel really a gzip file, not something else?
(like bz2 or such).

Samuel

Re: pvgrub boot problems

[M A Young]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 686 bytes --]

On Tue, 6 Apr 2010, Samuel Thibault wrote:

> M A Young, le Mon 05 Apr 2010 19:05:33 +0100, a écrit :
>> Yes, I have applied that and the patch you posted just before it and it
>> fixes the crash issue if there is no frame buffer. The boot still fails
>> for me because of the gzip issue but it didn't get that far previously
>> with this setup.
>
> The gzip issue?  Is your kernel really a gzip file, not something else?
> (like bz2 or such).

Yes, the older style for xen kernels, still used for some guests such as 
Red Hat Enterprise Linux 4 and 5, is to have a gzipped vmlinux executable. 
Xen support for bzimage style kernels is relatively recent.

 	Michael Young

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: pvgrub boot problems

[Samuel Thibault]

M A Young, le Tue 06 Apr 2010 10:55:30 +0100, a écrit :
> On Tue, 6 Apr 2010, Samuel Thibault wrote:
> 
> >M A Young, le Mon 05 Apr 2010 19:05:33 +0100, a écrit :
> >>Yes, I have applied that and the patch you posted just before it and it
> >>fixes the crash issue if there is no frame buffer. The boot still fails
> >>for me because of the gzip issue but it didn't get that far previously
> >>with this setup.
> >
> >The gzip issue?  Is your kernel really a gzip file, not something else?
> >(like bz2 or such).
> 
> Yes, the older style for xen kernels, still used for some guests such as 
> Red Hat Enterprise Linux 4 and 5, is to have a gzipped vmlinux executable. 
> Xen support for bzimage style kernels is relatively recent.

The problem with compression and crashes when using the fb should be
fixed by my latest patch (Fix crash on frontend shutdown failures)

Samuel

Re: pvgrub boot problems

[Jeremy Fitzhardinge]

On 04/06/2010 02:25 PM, Samuel Thibault wrote:
> M A Young, le Tue 06 Apr 2010 10:55:30 +0100, a écrit :
>    
>> On Tue, 6 Apr 2010, Samuel Thibault wrote:
>>
>>      
>>> M A Young, le Mon 05 Apr 2010 19:05:33 +0100, a écrit :
>>>        
>>>> Yes, I have applied that and the patch you posted just before it and it
>>>> fixes the crash issue if there is no frame buffer. The boot still fails
>>>> for me because of the gzip issue but it didn't get that far previously
>>>> with this setup.
>>>>          
>>> The gzip issue?  Is your kernel really a gzip file, not something else?
>>> (like bz2 or such).
>>>        
>> Yes, the older style for xen kernels, still used for some guests such as
>> Red Hat Enterprise Linux 4 and 5, is to have a gzipped vmlinux executable.
>> Xen support for bzimage style kernels is relatively recent.
>>      
> The problem with compression and crashes when using the fb should be
> fixed by my latest patch (Fix crash on frontend shutdown failures)
>    

I applied both, but something is still amiss.  Just commenting out the 
two frees allowed the domain to start, but the kernel started printing 
messages about the page state of various processes being bad.

After applying your two most recent patches (reverting the commented-out 
free_pages first), the domain seems to boot, but then hangs at some 
point early.

Disabling vfb still works fine.

     J

Re: pvgrub boot problems

[Samuel Thibault]

Jeremy Fitzhardinge, le Tue 06 Apr 2010 15:19:03 -0700, a écrit :
> I applied both, but something is still amiss.  Just commenting out the 
> two frees allowed the domain to start, but the kernel started printing 
> messages about the page state of various processes being bad.
> 
> After applying your two most recent patches (reverting the commented-out 
> free_pages first), the domain seems to boot, but then hangs at some 
> point early.
> 
> Disabling vfb still works fine.

Ok. The ground reason of all that is this:

shutdown_kbdfront: error changing state device/vkbd/0/state to 5: ENOENT

The MiniOS kbdfront driver can not stop the backend because xenstore
doesn't let it set the state to closing. I fail to understand how the
ENOENT error could happen (but not for the fb part!), and don't really
have the time to investigate further inside the meandres of xenstore...

Samuel

Re: pvgrub boot problems

[Jeremy Fitzhardinge]

On 04/06/2010 04:19 PM, Samuel Thibault wrote:
> Ok. The ground reason of all that is this:
>
> shutdown_kbdfront: error changing state device/vkbd/0/state to 5: ENOENT
>
> The MiniOS kbdfront driver can not stop the backend because xenstore
> doesn't let it set the state to closing. I fail to understand how the
> ENOENT error could happen (but not for the fb part!), and don't really
> have the time to investigate further inside the meandres of xenstore...
>    

Well, a solution that would work fine for me is if pvgrub could be told 
to ignore the pvfb. I find it pretty annoying to have to switch to a vnc 
viewer to boot, and it would be easier to do it on the xen console 
(though I can see that wouldn't be universally true).

     J

Re: pvgrub boot problems

[Samuel Thibault]

Jeremy Fitzhardinge, le Tue 06 Apr 2010 16:35:50 -0700, a écrit :
> Well, a solution that would work fine for me is if pvgrub could be told 
> to ignore the pvfb.

Err, it does it by default, _you_ told pvgrub to not ignore the pvfb by
leaving the splashscreen command :)

Samuel

Re: pvgrub boot problems

[Jeremy Fitzhardinge]

On 04/06/2010 04:43 PM, Samuel Thibault wrote:
> Jeremy Fitzhardinge, le Tue 06 Apr 2010 16:35:50 -0700, a écrit :
>    
>> Well, a solution that would work fine for me is if pvgrub could be told
>> to ignore the pvfb.
>>      
> Err, it does it by default, _you_ told pvgrub to not ignore the pvfb by
> leaving the splashscreen command :)
>    

Oh, OK.  I didn't realize that was a side-effect.

     J

Re: pvgrub boot problems

[Samuel Thibault]

Samuel Thibault, le Wed 07 Apr 2010 01:19:37 +0200, a écrit :
> I fail to understand how the ENOENT error could happen (but not for
> the fb part!),

Ok, grep told me ENOENT not only means about paths in the xenstore, but
also about transactions.  The patch I've submitted should fix that.

Samuel

Re: Xen 4.0.0-rc8 problem: xenconsole: Could not read tty from store: No such file or directory

[Keir Fraser]

On 27/03/2010 10:53, "Pasi Kärkkäinen" <pasik@iki.fi> wrote:

> And obviously I didn't get to see pygrub or select the domU kernel
> because the console connection failed.. domU did start up fine.
> 
> dom0 is Fedora 12 with Xen 4.0.0-rc8 (from Michael's rpm).
> 
> So what should I try to debug it?

Other thoughts: Try running 'xm console' later and see if the tty eventually
appears in xenstore. Look in xenstore and see if the tty entry appears to be
there (I think it's /local/domain/<domid>/serial/0/tty). Check whether
xenconsoled is running. Check syslog for consoled warnings/errors.

 -- Keir

Re: Xen 4.0.0-rc8 problem: xenconsole: Could not read tty from store: No such file or directory

[Pasi Kärkkäinen]

On Sat, Mar 27, 2010 at 11:25:42AM +0000, Keir Fraser wrote:
> On 27/03/2010 10:53, "Pasi Kärkkäinen" <pasik@iki.fi> wrote:
> 
> > And obviously I didn't get to see pygrub or select the domU kernel
> > because the console connection failed.. domU did start up fine.
> > 
> > dom0 is Fedora 12 with Xen 4.0.0-rc8 (from Michael's rpm).
> > 
> > So what should I try to debug it?
> 
> Other thoughts: Try running 'xm console' later

"xm console" works when I run it manually after "xm create -c" fails and returns.

> and see if the tty eventually appears in xenstore. 
> Look in xenstore and see if the tty entry appears to be
> there (I think it's /local/domain/<domid>/serial/0/tty). 

[root@dom0 ]# xenstore-ls /local/domain/2/serial
0 = ""
 tty = "/dev/pts/0"


> Check whether xenconsoled is running. 

xenconsoled is running. "xm console" manually works.

> Check syslog for consoled warnings/errors.
> 

No errors in /var/log/syslog or "xm log".

-- Pasi

Re: Xen 4.0.0-rc8 problem: xenconsole: Could not read tty from store: No such file or directory

[Keir Fraser]

On 27/03/2010 11:40, "Pasi Kärkkäinen" <pasik@iki.fi> wrote:

>>> So what should I try to debug it?
>> 
>> Other thoughts: Try running 'xm console' later
> 
> "xm console" works when I run it manually after "xm create -c" fails and
> returns.

Maybe the xenstore entry is slow to get set up for some reason. I'm not sure
how long 'xm create -c' waits around for.

 -- Keir

Re: Xen 4.0.0-rc8 problem: xenconsole: Could not read tty from store: No such file or directory

[Keir Fraser]

On 27/03/2010 11:48, "Keir Fraser" <keir.fraser@eu.citrix.com> wrote:

>>>> So what should I try to debug it?
>>> 
>>> Other thoughts: Try running 'xm console' later
>> 
>> "xm console" works when I run it manually after "xm create -c" fails and
>> returns.
> 
> Maybe the xenstore entry is slow to get set up for some reason. I'm not sure
> how long 'xm create -c' waits around for.

Again also could point at pygrub, since presumably that affects console
handling (must hijack the guest console to print boot menu and stuff?).

 -- Keir

Re: Xen 4.0.0-rc8 problem: xenconsole: Could not read tty from store: No such file or directory

[Pasi Kärkkäinen]

On Sat, Mar 27, 2010 at 07:56:00AM +0000, Keir Fraser wrote:
> I'm able to create HVM and PV domains okay. Are you saying that specifically
> xend 'managed domain' functionality is broken? Any idea if that is a recent
> regression, if so?
> 

Michael might be able to comment if this is recent or not.. 
I'll try older rcs..

-- Pasi

> 
> On 26/03/2010 20:35, "Pasi Kärkkäinen" <pasik@iki.fi> wrote:
> 
> > Hello,
> > 
> > Michael Young noticed this problem, and I can confirm it:
> > 
> > # xm start f12test -c
> > xenconsole: Could not read tty from store: No such file or directory
> > 
> > -- Pasi
> > 
> > 
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
> 
>