Re: Debian Kernel: Xen-fb-frontend as a module?

Re: Debian Kernel: Xen-fb-frontend as a module?

[Ian Campbell]

create !
title it "30s delay loading xenfb driver on some systems"
owner it Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
thanks

Hi James,

Some of the other Xen devs were discussing an issue which sounded
awfully similar to this one, so I am copying the xen-devel list and
creating a Xen bug to track the issue, please CC xen-devel (no need to
subscribe but you may be moderated the first time), since the tracker
slurps mails from the list.

I'm not sure of the details of the other issue but it involved
http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=b1a3b1c8a8d963424c4699efa64dd8986b2f76d7 hopefully Konrad or one of the others will follow up.

For xen-devel, the first two mails in this thread are
https://lists.debian.org/debian-kernel/2014/09/msg00229.html and 
https://lists.debian.org/debian-kernel/2014/09/msg00233.html

Ian.

On Tue, 2014-09-23 at 21:07 +0800, James Bromberger wrote:
> (Sorry for the delay, was offlist and didnt see your response Ian, so have now subscribed for this discussion)
> 
> 
> > Isn't the delay actually because the host has exposed a device/vfb/0 to
> > the guest apparently without providing the corresponding backend device?
> 
> Sorry, I'm not aware. All I see is that several distros are removing this 
> module from their initrd and blacklisting it when booting in an HVM 
> environment on EC2.
> 
> 
> 
> > > [   30.969632] xenbus_probe_frontend: Timeout connecting to device:
> > > device/vfb/0 (local state 3, remote state 1)
> 
> > This means that the frontend has reached XenbusStateInitialised (3)
> > while the backend is not responding and has stayed in
> > XenbusStateInitialising (1).
> 
> > I don't know what the chances of getting the host side fixed here, maybe
> > we have to live with some sort of hack in the frontend (or live with the
> > delay). We should be careful not to break things for people with
> > correctly configured hosts though (which might rule out making it
> > modular, not sure).
> 
> 
> Agreed. I am reluctant to add another kernel binary package for EC2 (but 
> perhaps in such a beast we could bump the ixgbevf driver revision
> at http://sourceforge.net/projects/e1000/files/ixgbevf%20stable/ from 
> 2.12.1 to the EC2 recommended 2.14.1 or newer...).
> 
> 
> > FWIW it looks like the upstream kernel already special cases missing
> > pvfb and only waits 30s instead of the 270s it would wait for a more
> > critical device like a disk, so it could be worse!
> 
> > Perhaps upstream would consider a patch which adds a command line option
> > listing devices to ignore/not wait for?
> 
> The status-quo 30 sec wait is perhaps the least disruptive option, but as 
> other distros are able to boot to user space in around 5 seconds (incl 
> Ubuntu), Debian Jessie comes in at 33 seconds due to this. 
> 
> 
> The patch idea sounds nice (but beyond me to contribute). Perhaps an 
> easier patch is to customise the timer to wait for this device 
> (xen_fb_frontend_timeout=1)?
> 
> Thanks
> 
>   James
> 
> 
> 

Re: Debian Kernel: Xen-fb-frontend as a module?

[Ian Campbell]

create ^
title it "30s delay loading xenfb driver on some systems"
owner it Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
thanks

>From address which is allowed to create bugs this time...

Ian.

Processed: Re: Debian Kernel: Xen-fb-frontend as a module?

[xen]

Processing commands for xen@bugs.xenproject.org:

> create ^
Created new bug #43 rooted at `<1411482601.1781.56.camel@kazak.uk.xensource.com>'
Title: `Re: [Xen-devel] Debian Kernel: Xen-fb-frontend as a module?'
> title it "30s delay loading xenfb driver on some systems"
Set title for #43 to `"30s delay loading xenfb driver on some systems"'
> owner it Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Change owner for #43 to `Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>'
> thanks
Finished processing.

Modified/created Bugs:
 - 43: http://bugs.xenproject.org/xen/bug/43 (new)

---
Xen Hypervisor Bug Tracker
See http://wiki.xen.org/wiki/Reporting_Bugs_against_Xen for information on reporting bugs
Contact xen-bugs-owner@bugs.xenproject.org with any infrastructure issues

Re: Debian Kernel: Xen-fb-frontend as a module?

[David Vrabel]

On 23/09/14 15:30, Ian Campbell wrote:
> create !
> title it "30s delay loading xenfb driver on some systems"
> owner it Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
> thanks
> 
> Hi James,
> 
> Some of the other Xen devs were discussing an issue which sounded
> awfully similar to this one, so I am copying the xen-devel list and
> creating a Xen bug to track the issue, please CC xen-devel (no need to
> subscribe but you may be moderated the first time), since the tracker
> slurps mails from the list.
> 
> I'm not sure of the details of the other issue but it involved
> http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=b1a3b1c8a8d963424c4699efa64dd8986b2f76d7 hopefully Konrad or one of the others will follow up.
> 
> For xen-devel, the first two mails in this thread are
> https://lists.debian.org/debian-kernel/2014/09/msg00229.html and 
> https://lists.debian.org/debian-kernel/2014/09/msg00233.html

The wait stuff for xenbus devices looks like pre-dates distros handling
asynchronous devices with suitable initrds etc.

I think we need a command line configurable white list of device types
to wait for.  The default should be (to match what was done historically):

PV: vbd, vif, pci, vfb, vkbd.
HVM: vbd, vif.

I also think it should be possible to default (via a config option) to
an empty white list.

David

Re: Debian Kernel: Xen-fb-frontend as a module?

[Ian Campbell]

On Tue, 2014-09-23 at 16:43 +0100, David Vrabel wrote:
> On 23/09/14 15:30, Ian Campbell wrote:
> > create !
> > title it "30s delay loading xenfb driver on some systems"
> > owner it Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
> > thanks
> > 
> > Hi James,
> > 
> > Some of the other Xen devs were discussing an issue which sounded
> > awfully similar to this one, so I am copying the xen-devel list and
> > creating a Xen bug to track the issue, please CC xen-devel (no need to
> > subscribe but you may be moderated the first time), since the tracker
> > slurps mails from the list.
> > 
> > I'm not sure of the details of the other issue but it involved
> > http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=b1a3b1c8a8d963424c4699efa64dd8986b2f76d7 hopefully Konrad or one of the others will follow up.
> > 
> > For xen-devel, the first two mails in this thread are
> > https://lists.debian.org/debian-kernel/2014/09/msg00229.html and 
> > https://lists.debian.org/debian-kernel/2014/09/msg00233.html
> 
> The wait stuff for xenbus devices looks like pre-dates distros handling
> asynchronous devices with suitable initrds etc.

Perhaps, but I think most distros still don't use rootwait by default so
unless / turns up reasonably promptly (single digit seconds) the boot
may fail.

> I think we need a command line configurable white list of device types
> to wait for.  The default should be (to match what was done historically):
> 
> PV: vbd, vif, pci, vfb, vkbd.
> HVM: vbd, vif.
> 
> I also think it should be possible to default (via a config option) to
> an empty white list.

Irrespective of the above this seems like a reasonable enough idea to
me.

Ian.

Re: Debian Kernel: Xen-fb-frontend as a module?

[Konrad Rzeszutek Wilk]

On Tue, Sep 23, 2014 at 04:57:26PM +0100, Ian Campbell wrote:
> On Tue, 2014-09-23 at 16:43 +0100, David Vrabel wrote:
> > On 23/09/14 15:30, Ian Campbell wrote:
> > > create !
> > > title it "30s delay loading xenfb driver on some systems"
> > > owner it Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
> > > thanks
> > > 
> > > Hi James,
> > > 
> > > Some of the other Xen devs were discussing an issue which sounded
> > > awfully similar to this one, so I am copying the xen-devel list and
> > > creating a Xen bug to track the issue, please CC xen-devel (no need to
> > > subscribe but you may be moderated the first time), since the tracker
> > > slurps mails from the list.
> > > 
> > > I'm not sure of the details of the other issue but it involved
> > > http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=b1a3b1c8a8d963424c4699efa64dd8986b2f76d7 hopefully Konrad or one of the others will follow up.

That was with an HVM guest running under Xen 4.1 in which this
guest config was used:

vfb = ['type=vnc,vncunused=1,vnclisten=0.0.0.0']

Xend would create an XenStore keys for the PV framebuffer and also making
sure that QEMU VGA driver was running. The end result was that the guest
would boot up to Xorg VGA driver, but the frame buffer console (so from
the moment GRUB2 started Linux up to Xorg) would try to use the xen-fbfront.

And since this is HVM guest and VNCviewer was slurping contents from the
VGA buffer - which was not used at all - we wouldn't get anything.

Reverting the above patch fixed the issue.
> > > 
> > > For xen-devel, the first two mails in this thread are
> > > https://lists.debian.org/debian-kernel/2014/09/msg00229.html and 
> > > https://lists.debian.org/debian-kernel/2014/09/msg00233.html
> > 
> > The wait stuff for xenbus devices looks like pre-dates distros handling
> > asynchronous devices with suitable initrds etc.
> 
> Perhaps, but I think most distros still don't use rootwait by default so
> unless / turns up reasonably promptly (single digit seconds) the boot
> may fail.
> 
> > I think we need a command line configurable white list of device types
> > to wait for.  The default should be (to match what was done historically):
> > 
> > PV: vbd, vif, pci, vfb, vkbd.
> > HVM: vbd, vif.
> > 
> > I also think it should be possible to default (via a config option) to
> > an empty white list.
> 
> Irrespective of the above this seems like a reasonable enough idea to
> me.

What about ARM?
> 
> Ian.
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: Debian Kernel: Xen-fb-frontend as a module?

[Stefano Stabellini]

On Tue, 23 Sep 2014, Konrad Rzeszutek Wilk wrote:
> On Tue, Sep 23, 2014 at 04:57:26PM +0100, Ian Campbell wrote:
> > On Tue, 2014-09-23 at 16:43 +0100, David Vrabel wrote:
> > > On 23/09/14 15:30, Ian Campbell wrote:
> > > > create !
> > > > title it "30s delay loading xenfb driver on some systems"
> > > > owner it Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
> > > > thanks
> > > > 
> > > > Hi James,
> > > > 
> > > > Some of the other Xen devs were discussing an issue which sounded
> > > > awfully similar to this one, so I am copying the xen-devel list and
> > > > creating a Xen bug to track the issue, please CC xen-devel (no need to
> > > > subscribe but you may be moderated the first time), since the tracker
> > > > slurps mails from the list.
> > > > 
> > > > I'm not sure of the details of the other issue but it involved
> > > > http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=b1a3b1c8a8d963424c4699efa64dd8986b2f76d7 hopefully Konrad or one of the others will follow up.
> 
> That was with an HVM guest running under Xen 4.1 in which this
> guest config was used:
> 
> vfb = ['type=vnc,vncunused=1,vnclisten=0.0.0.0']
> 
> Xend would create an XenStore keys for the PV framebuffer and also making
> sure that QEMU VGA driver was running. The end result was that the guest
> would boot up to Xorg VGA driver, but the frame buffer console (so from
> the moment GRUB2 started Linux up to Xorg) would try to use the xen-fbfront.
> 
> And since this is HVM guest and VNCviewer was slurping contents from the
> VGA buffer - which was not used at all - we wouldn't get anything.
> 
> Reverting the above patch fixed the issue.

If you have a vfb line in your config file, aren't you actually
explicitly requesting a vfb frontend/backend pair?
XenD is just doing what the user asked him to do, I wouldn't call this a
bug.

What is strange is that given that there is no running vfb backend for
HVM guests in a regular Xen 4.1 installation, xen-fbfront could never
reach the "connected" state ("4" on xenstore): so why is Linux trying to
use vfb when the frontend is not even connected?

The reason is that xenfb_probe calls register_framebuffer and
xenfb_make_preferred_console too early, before even knowing if the
backend is alive.

I would move the register_framebuffer and xenfb_make_preferred_console
calls in xenfb_backend_changed, case XenbusStateConnected. That should
fix it.



> > > > 
> > > > For xen-devel, the first two mails in this thread are
> > > > https://lists.debian.org/debian-kernel/2014/09/msg00229.html and 
> > > > https://lists.debian.org/debian-kernel/2014/09/msg00233.html
> > > 
> > > The wait stuff for xenbus devices looks like pre-dates distros handling
> > > asynchronous devices with suitable initrds etc.
> > 
> > Perhaps, but I think most distros still don't use rootwait by default so
> > unless / turns up reasonably promptly (single digit seconds) the boot
> > may fail.
> > 
> > > I think we need a command line configurable white list of device types
> > > to wait for.  The default should be (to match what was done historically):
> > > 
> > > PV: vbd, vif, pci, vfb, vkbd.
> > > HVM: vbd, vif.
> > > 
> > > I also think it should be possible to default (via a config option) to
> > > an empty white list.
> > 
> > Irrespective of the above this seems like a reasonable enough idea to
> > me.
> 
> What about ARM?

On ARM (and on PV on HVM x86) we want vfb to work by default if the
xenstore keys are there.  But I don't think that it means we need to
wait 30 secs for it at boot.  What is a reasonable amount of time to
wait for on a slow and overcommitted system? Maybe 5 to 10 secs?

Re: Debian Kernel: Xen-fb-frontend as a module?

[David Vrabel]

On 24/09/14 11:23, Stefano Stabellini wrote:
> On Tue, 23 Sep 2014, Konrad Rzeszutek Wilk wrote:
>> On Tue, Sep 23, 2014 at 04:57:26PM +0100, Ian Campbell wrote:
>>> On Tue, 2014-09-23 at 16:43 +0100, David Vrabel wrote:
>>>> On 23/09/14 15:30, Ian Campbell wrote:
>>>>> create !
>>>>> title it "30s delay loading xenfb driver on some systems"
>>>>> owner it Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
>>>>> thanks
>>>>>
>>>>> Hi James,
>>>>>
>>>>> Some of the other Xen devs were discussing an issue which sounded
>>>>> awfully similar to this one, so I am copying the xen-devel list and
>>>>> creating a Xen bug to track the issue, please CC xen-devel (no need to
>>>>> subscribe but you may be moderated the first time), since the tracker
>>>>> slurps mails from the list.
>>>>>
>>>>> I'm not sure of the details of the other issue but it involved
>>>>> http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=b1a3b1c8a8d963424c4699efa64dd8986b2f76d7 hopefully Konrad or one of the others will follow up.
>>
>> That was with an HVM guest running under Xen 4.1 in which this
>> guest config was used:
>>
>> vfb = ['type=vnc,vncunused=1,vnclisten=0.0.0.0']
>>
>> Xend would create an XenStore keys for the PV framebuffer and also making
>> sure that QEMU VGA driver was running. The end result was that the guest
>> would boot up to Xorg VGA driver, but the frame buffer console (so from
>> the moment GRUB2 started Linux up to Xorg) would try to use the xen-fbfront.
>>
>> And since this is HVM guest and VNCviewer was slurping contents from the
>> VGA buffer - which was not used at all - we wouldn't get anything.
>>
>> Reverting the above patch fixed the issue.
> 
> If you have a vfb line in your config file, aren't you actually
> explicitly requesting a vfb frontend/backend pair?
> XenD is just doing what the user asked him to do, I wouldn't call this a
> bug.
> 
> What is strange is that given that there is no running vfb backend for
> HVM guests in a regular Xen 4.1 installation, xen-fbfront could never
> reach the "connected" state ("4" on xenstore): so why is Linux trying to
> use vfb when the frontend is not even connected?
> 
> The reason is that xenfb_probe calls register_framebuffer and
> xenfb_make_preferred_console too early, before even knowing if the
> backend is alive.
> 
> I would move the register_framebuffer and xenfb_make_preferred_console
> calls in xenfb_backend_changed, case XenbusStateConnected. That should
> fix it.

I don't think that will help.  Registering the frontend driver will
still wait for CONNECTED, regardless of whether the frame buffer will
actually be used.

David

Re: Debian Kernel: Xen-fb-frontend as a module?

[Stefano Stabellini]

On Wed, 24 Sep 2014, David Vrabel wrote:
> On 24/09/14 11:23, Stefano Stabellini wrote:
> > On Tue, 23 Sep 2014, Konrad Rzeszutek Wilk wrote:
> >> On Tue, Sep 23, 2014 at 04:57:26PM +0100, Ian Campbell wrote:
> >>> On Tue, 2014-09-23 at 16:43 +0100, David Vrabel wrote:
> >>>> On 23/09/14 15:30, Ian Campbell wrote:
> >>>>> create !
> >>>>> title it "30s delay loading xenfb driver on some systems"
> >>>>> owner it Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
> >>>>> thanks
> >>>>>
> >>>>> Hi James,
> >>>>>
> >>>>> Some of the other Xen devs were discussing an issue which sounded
> >>>>> awfully similar to this one, so I am copying the xen-devel list and
> >>>>> creating a Xen bug to track the issue, please CC xen-devel (no need to
> >>>>> subscribe but you may be moderated the first time), since the tracker
> >>>>> slurps mails from the list.
> >>>>>
> >>>>> I'm not sure of the details of the other issue but it involved
> >>>>> http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=b1a3b1c8a8d963424c4699efa64dd8986b2f76d7 hopefully Konrad or one of the others will follow up.
> >>
> >> That was with an HVM guest running under Xen 4.1 in which this
> >> guest config was used:
> >>
> >> vfb = ['type=vnc,vncunused=1,vnclisten=0.0.0.0']
> >>
> >> Xend would create an XenStore keys for the PV framebuffer and also making
> >> sure that QEMU VGA driver was running. The end result was that the guest
> >> would boot up to Xorg VGA driver, but the frame buffer console (so from
> >> the moment GRUB2 started Linux up to Xorg) would try to use the xen-fbfront.
> >>
> >> And since this is HVM guest and VNCviewer was slurping contents from the
> >> VGA buffer - which was not used at all - we wouldn't get anything.
> >>
> >> Reverting the above patch fixed the issue.
> > 
> > If you have a vfb line in your config file, aren't you actually
> > explicitly requesting a vfb frontend/backend pair?
> > XenD is just doing what the user asked him to do, I wouldn't call this a
> > bug.
> > 
> > What is strange is that given that there is no running vfb backend for
> > HVM guests in a regular Xen 4.1 installation, xen-fbfront could never
> > reach the "connected" state ("4" on xenstore): so why is Linux trying to
> > use vfb when the frontend is not even connected?
> > 
> > The reason is that xenfb_probe calls register_framebuffer and
> > xenfb_make_preferred_console too early, before even knowing if the
> > backend is alive.
> > 
> > I would move the register_framebuffer and xenfb_make_preferred_console
> > calls in xenfb_backend_changed, case XenbusStateConnected. That should
> > fix it.
> 
> I don't think that will help.  Registering the frontend driver will
> still wait for CONNECTED, regardless of whether the frame buffer will
> actually be used.

It will wait for a long time but then it should work.
The problem of the long wait time is a different matter. To solve that
couldn't we simply reduce the wait time?

Re: Debian Kernel: Xen-fb-frontend as a module?

[Ian Campbell]

On Wed, 2014-09-24 at 11:23 +0100, Stefano Stabellini wrote:

> On ARM (and on PV on HVM x86) we want vfb to work by default if the
> xenstore keys are there.  But I don't think that it means we need to
> wait 30 secs for it at boot.  What is a reasonable amount of time to
> wait for on a slow and overcommitted system? Maybe 5 to 10 secs?

NB there are already two classes of wait. Non-essential
(xenbus_probe_frontend.c's wording) wait 30s essential wait for an
additional 270s (so 300s total).

Non-essential appears to be vfb and vkbd (but not mouse?). We could
certainly consider reducing the 30 (and increasing the 270 to
correspond).

Ian.