[PATCH (v2,resend) 0/3] Rework the devshell

[PATCH (v2,resend) 0/3] Rework the devshell

[Chris Larson]

From: Chris Larson <chris_larson@mentor.com>

- Adds a utility metaclass for maintaining class registries
- Shifts the handling of spawning a terminal into a new 'terminal' python
 module
- Automatically selects an appropriate terminal based on what you have
 available on your machine
- Still allows manual selection, but in a more convenient way, and falls back
 to auto if you select an invalid or unavailable (not installed) terminal
- Improves how we spawn screen:

 - Shows a warning when spawning it, so do_devshell doesn't appear to hang
 - Spawns a named screen session 'devshell', to avoid confusion when the user
   already has other screen sessions running

Example usage of the new API for others (e.g. patch resolver) to use:

    import oe.terminal
    oe.terminal.run(command, terminal_title, d)

Pull URL: git://git.openembedded.org/openembedded-core-contrib
  Branch: kergoth/devshell
  Browse: http://git.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=kergoth/devshell

Thanks,
    Chris Larson <chris_larson@mentor.com>
---


Chris Larson (3):
  oe.classutils: add module
  Rework how the devshell functions
  oe.terminal: improve how we spawn screen

 meta/classes/devshell.bbclass |   25 ++------
 meta/conf/bitbake.conf        |    9 +++
 meta/lib/oe/classutils.py     |   45 ++++++++++++++
 meta/lib/oe/terminal.py       |  131 +++++++++++++++++++++++++++++++++++++++++
 4 files changed, 191 insertions(+), 19 deletions(-)
 create mode 100644 meta/lib/oe/classutils.py
 create mode 100644 meta/lib/oe/terminal.py

[PATCH (v2,resend) 1/3] oe.classutils: add module

[Chris Larson]

From: Chris Larson <chris_larson@mentor.com>

This adds a ClassRegistry utility metaclass, as maintaining a class registry
is a fairly common thing to do.

Signed-off-by: Chris Larson <chris_larson@mentor.com>
---
 meta/lib/oe/classutils.py |   43 +++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 43 insertions(+), 0 deletions(-)
 create mode 100644 meta/lib/oe/classutils.py

diff --git a/meta/lib/oe/classutils.py b/meta/lib/oe/classutils.py
new file mode 100644
index 0000000..58188fd
--- /dev/null
+++ b/meta/lib/oe/classutils.py
@@ -0,0 +1,43 @@
+class ClassRegistry(type):
+    """Maintain a registry of classes, indexed by name.
+
+Note that this implementation requires that the names be unique, as it uses
+a dictionary to hold the classes by name.
+
+The name in the registry can be overridden via the 'name' attribute of the
+class, and the 'priority' attribute controls priority. The prioritized()
+method returns the registered classes in priority order.
+
+Subclasses of ClassRegistry may define an 'implemented' property to exert
+control over whether the class will be added to the registry (e.g. to keep
+abstract base classes out of the registry)."""
+    priority = 0
+    class __metaclass__(type):
+        """Give each ClassRegistry their own registry"""
+        def __init__(cls, name, bases, attrs):
+            cls.registry = {}
+            type.__init__(cls, name, bases, attrs)
+
+    def __init__(cls, name, bases, attrs):
+        super(ClassRegistry, cls).__init__(name, bases, attrs)
+        try:
+            if not cls.implemented:
+                return
+        except AttributeError:
+            pass
+
+        try:
+            cls.name
+        except AttributeError:
+            cls.name = name
+        cls.registry[cls.name] = cls
+
+    @classmethod
+    def prioritized(tcls):
+        return sorted(tcls.registry.values(),
+                      key=lambda v: v.priority, reverse=True)
+
+    def unregister(cls):
+        for key in cls.registry.keys():
+            if cls.registry[key] is cls:
+                del cls.registry[key]
-- 
1.7.1

[PATCH (v2, resend) 2/3] Rework how the devshell functions

[Chris Larson]

From: Chris Larson <chris_larson@mentor.com>

In the new implementation, each known terminal is defined as a class in
oe.terminal, as a subclass of bb.process.Popen. It obeys the OE_TERMINAL
variable, which is a 'choice' typed variable. This variable may be 'auto',
'none', or any of the names of the defined terminals.

When using 'auto', or requesting an unsupported terminal, we attempt to spawn
them in priority order until we get one that's available on this system (and
in the case of the X terminals, has DISPLAY defined).  The 'none' value is
used when we're doing things like automated builds, and want to ensure that no
terminal is *ever* spawned, under any circumstances.

Current available terminals:

    gnome
    konsole
    xterm
    rxvt
    screen

Signed-off-by: Chris Larson <chris_larson@mentor.com>
---
 meta/classes/devshell.bbclass |   25 ++------
 meta/conf/bitbake.conf        |    9 +++
 meta/lib/oe/classutils.py     |    2 +
 meta/lib/oe/terminal.py       |  126 +++++++++++++++++++++++++++++++++++++++++
 4 files changed, 143 insertions(+), 19 deletions(-)
 create mode 100644 meta/lib/oe/terminal.py

diff --git a/meta/classes/devshell.bbclass b/meta/classes/devshell.bbclass
index 5f262f4..216a9dc 100644
--- a/meta/classes/devshell.bbclass
+++ b/meta/classes/devshell.bbclass
@@ -1,22 +1,9 @@
-do_devshell[dirs] = "${S}"
-do_devshell[nostamp] = "1"
-
-XAUTHORITY ?= "${HOME}/.Xauthority"
-
-devshell_do_devshell() {
-	export DISPLAY='${DISPLAY}'
-	export DBUS_SESSION_BUS_ADDRESS='${DBUS_SESSION_BUS_ADDRESS}'
-	export XAUTHORITY='${XAUTHORITY}'
-	export TERMWINDOWTITLE="Bitbake Developer Shell"
-	export EXTRA_OEMAKE='${EXTRA_OEMAKE}'
-	export SHELLCMDS="bash"
-	${TERMCMDRUN}
-	if [ $? -ne 0 ]; then
-	    echo "Fatal: '${TERMCMD}' not found. Check TERMCMD variable."
-	    exit 1
-	fi
+python do_devshell () {
+    import oe.terminal
+    oe.terminal.run(d.getVar('SHELL', True), 'OpenEmbedded Developer Shell', d)
 }
-addtask devshell after do_patch
 
-EXPORT_FUNCTIONS do_devshell
+addtask devshell after do_patch
 
+do_devshell[dirs] = "${S}"
+do_devshell[nostamp] = "1"
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index a0af672..45a426d 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -539,6 +539,15 @@ PREFERRED_PROVIDER_virtual/fakeroot-native ?= "pseudo-native"
 # UI/Interaction Configuration
 ##################################################################
 
+
+OE_TERMINAL ?= 'auto'
+OE_TERMINAL[type] = 'choice'
+OE_TERMINAL[choices] = 'auto none \
+                        ${@" ".join(o.name \
+                                    for o in oe.terminal.prioritized())}'
+
+export XAUTHORITY ?= "${HOME}/.Xauthority"
+export SHELL ?= "bash"
 export SHELLCMDS = "bash"
 # Some common terminal programs to choose from
 GNOME_TERMCMD = 'gnome-terminal --disable-factory -t "$TERMWINDOWTITLE"'
diff --git a/meta/lib/oe/classutils.py b/meta/lib/oe/classutils.py
index 58188fd..42b0f49 100644
--- a/meta/lib/oe/classutils.py
+++ b/meta/lib/oe/classutils.py
@@ -1,3 +1,5 @@
+__author__ = 'kergoth'
+
 class ClassRegistry(type):
     """Maintain a registry of classes, indexed by name.
 
diff --git a/meta/lib/oe/terminal.py b/meta/lib/oe/terminal.py
new file mode 100644
index 0000000..e14fc5d
--- /dev/null
+++ b/meta/lib/oe/terminal.py
@@ -0,0 +1,126 @@
+import logging
+import os
+import oe.classutils
+import oe.data
+import shlex
+import bb
+from bb.process import Popen, ExecutionError
+
+
+logger = logging.getLogger('BitBake.OE.Terminal')
+
+
+class UnsupportedTerminal(StandardError):
+    pass
+
+class NoSupportedTerminals(StandardError):
+    pass
+
+
+class Registry(oe.classutils.ClassRegistry):
+    command = None
+
+    def __init__(cls, name, bases, attrs):
+        super(Registry, cls).__init__(name.lower(), bases, attrs)
+
+    @property
+    def implemented(cls):
+        return bool(cls.command)
+
+
+class Terminal(Popen):
+    __metaclass__ = Registry
+
+    def __init__(self, command, title=None):
+        self.format_command(command, title)
+        logger.debug(1, "%s: running %s", self.name, self.command)
+
+        try:
+            Popen.__init__(self, self.command, shell=False)
+        except OSError as exc:
+            import errno
+            if exc.errno == errno.ENOENT:
+                raise UnsupportedTerminal(self.name)
+            else:
+                raise
+
+    def format_command(self, command, title):
+        fmt = {'title': title or 'Terminal', 'command': command}
+        if isinstance(self.command, basestring):
+            self.command = shlex.split(self.command.format(**fmt))
+        else:
+            self.command = [element.format(**fmt) for element in self.command]
+
+class XTerminal(Terminal):
+    def __init__(self, command, title=None):
+        Terminal.__init__(self, command, title)
+        if not os.environ.get('DISPLAY'):
+            raise UnsupportedTerminal(self.name)
+
+class Gnome(XTerminal):
+    command = 'gnome-terminal --disable-factory -t "{title}" -x {command}'
+    priority = 2
+
+class Konsole(XTerminal):
+    command = 'konsole -T "{title}" -e {command}'
+    priority = 2
+
+class XTerm(XTerminal):
+    command = 'xterm -T "{title}" -e {command}'
+    priority = 1
+
+class Rxvt(XTerminal):
+    command = 'rxvt -T "{title}" -e {command}'
+    priority = 1
+
+class Screen(Terminal):
+    command = 'screen -D -m -t "{title}" {command}'
+
+
+def prioritized():
+    return Registry.prioritized()
+
+def run(command, title, d):
+    terminal = oe.data.typed_value('OE_TERMINAL', d).lower()
+    if terminal == 'none':
+        bb.fatal('Devshell usage disabled with OE_TERMINAL')
+    elif terminal != 'auto':
+        try:
+            spawn(terminal, command, title)
+            return
+        except UnsupportedTerminal:
+            bb.warn('Unsupported terminal "%s", defaulting to "auto"' %
+                    terminal)
+        except ExecutionError as exc:
+            bb.fatal('Unable to spawn terminal %s: %s' % (terminal, exc))
+
+    try:
+        spawn_preferred(command, title)
+    except NoSupportedTerminals:
+        bb.fatal('No valid terminal found, unable to open devshell')
+    except ExecutionError as exc:
+        bb.fatal('Unable to spawn terminal %s: %s' % (terminal, exc))
+
+def spawn_preferred(command, title=None):
+    """Spawn the first supported terminal, by priority"""
+    for terminal in prioritized():
+        try:
+            spawn(terminal.name, command, title)
+            break
+        except UnsupportedTerminal:
+            continue
+    else:
+        raise NoSupportedTerminals()
+
+def spawn(name, command, title=None):
+    """Spawn the specified terminal, by name"""
+    logger.debug(1, 'Attempting to spawn terminal "%s"', name)
+    try:
+        terminal = Registry.registry[name]
+    except KeyError:
+        raise UnsupportedTerminal(name)
+
+    pipe = terminal(command, title)
+    output = pipe.communicate()[0]
+    if pipe.returncode != 0:
+        raise ExecutionError(pipe.command, pipe.returncode, output)
-- 
1.7.1

[PATCH (v2, resend) 3/3] oe.terminal: improve how we spawn screen

[Chris Larson]

From: Chris Larson <chris_larson@mentor.com>

- Name the screen session 'devshell', to avoid confusion if running bitbake
  itself under a screen session.
- Display a warning message when spawning screen, so it's clear to the user
  that screen has been run (otherwise do_devshell just appears to hang).

Signed-off-by: Chris Larson <chris_larson@mentor.com>
---
 meta/lib/oe/terminal.py |    7 ++++++-
 1 files changed, 6 insertions(+), 1 deletions(-)

diff --git a/meta/lib/oe/terminal.py b/meta/lib/oe/terminal.py
index e14fc5d..4fd281c 100644
--- a/meta/lib/oe/terminal.py
+++ b/meta/lib/oe/terminal.py
@@ -74,7 +74,12 @@ class Rxvt(XTerminal):
     priority = 1
 
 class Screen(Terminal):
-    command = 'screen -D -m -t "{title}" {command}'
+    command = 'screen -D -m -t "{title}" -S devshell {command}'
+
+    def __init__(self, command, title=None):
+        Terminal.__init__(self, command, title)
+        logger.warn('Screen started. Please connect in another terminal with '
+                    '"screen -r devshell"')
 
 
 def prioritized():
-- 
1.7.1

Re: [PATCH (v2,resend) 0/3] Rework the devshell

[Joshua Lock]

On Wed, 2011-05-18 at 10:17 -0700, Chris Larson wrote:
> From: Chris Larson <chris_larson@mentor.com>
> 
> - Adds a utility metaclass for maintaining class registries
> - Shifts the handling of spawning a terminal into a new 'terminal' python
>  module
> - Automatically selects an appropriate terminal based on what you have
>  available on your machine
> - Still allows manual selection, but in a more convenient way, and falls back
>  to auto if you select an invalid or unavailable (not installed) terminal
> - Improves how we spawn screen:
> 
>  - Shows a warning when spawning it, so do_devshell doesn't appear to hang
>  - Spawns a named screen session 'devshell', to avoid confusion when the user
>    already has other screen sessions running
> 
> Example usage of the new API for others (e.g. patch resolver) to use:
> 
>     import oe.terminal
>     oe.terminal.run(command, terminal_title, d)
> 
> Pull URL: git://git.openembedded.org/openembedded-core-contrib
>   Branch: kergoth/devshell
>   Browse: http://git.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=kergoth/devshell
> 
> Thanks,
>     Chris Larson <chris_larson@mentor.com>

I like this series!

Speaking of devshell, last I tried (and Paul E can correct me here)
Konsole doesn't wait when it's spawned so is currently pretty useless
for devshell (bitbake continues as if the devshell had finished and if a
patch hasn't applied....).

Perhaps we should remove support for it from the class? Or is there some
better way we can work around this?

As an aside: I think somewhere in meta-yocto, to maintain current
behaviour we'd want to set: OE_TERMINAL ?= 'Gnome' to maintain
gnome-terminal as the default (when installed).

Acked-by: Joshua Lock <josh@linux.intel.com>

Cheers,
Joshua

> ---
> 
> 
> Chris Larson (3):
>   oe.classutils: add module
>   Rework how the devshell functions
>   oe.terminal: improve how we spawn screen
> 
>  meta/classes/devshell.bbclass |   25 ++------
>  meta/conf/bitbake.conf        |    9 +++
>  meta/lib/oe/classutils.py     |   45 ++++++++++++++
>  meta/lib/oe/terminal.py       |  131 +++++++++++++++++++++++++++++++++++++++++
>  4 files changed, 191 insertions(+), 19 deletions(-)
>  create mode 100644 meta/lib/oe/classutils.py
>  create mode 100644 meta/lib/oe/terminal.py
> 
> 
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

-- 
Joshua Lock
        Yocto Project Build Monkey
        Intel Open Source Technology Centre

Re: [PATCH (v2,resend) 0/3] Rework the devshell

[Paul Eggleton]

On Friday 20 May 2011 01:02:58 Joshua Lock wrote:
> Speaking of devshell, last I tried (and Paul E can correct me here)
> Konsole doesn't wait when it's spawned so is currently pretty useless
> for devshell (bitbake continues as if the devshell had finished and if a
> patch hasn't applied....).

I can confirm this - right now our Konsole support just does not work and I 
imagine it has not since KDE 4.0. Adding the --nofork command line option 
which is supposed to make it wait does not work either, because if the 
controlling terminal is not interactive it assumes there is no other instance 
of Konsole running, tries to register a DBUS service and fails (and it's hard 
to imagine someone using KDE and running bitbake without having an instance of 
Konsole running already).
 
> Perhaps we should remove support for it from the class? Or is there some
> better way we can work around this?

A while ago I had a look adding a command line option to Konsole itself to 
make this work but didn't get very far, although I don't think it should be 
too difficult. There is a workaround script [1] but when I tried it just now I 
found it again has problems if the controlling terminal is not interactive 
(never returns). Frankly the script is hideous and maybe it was broken in a 
version of KDE since the script was written, I don't know.

It might be prudent to remove the Konsole support until it has been fixed 
appropriately; at the very least it should be documented to only work with 
Konsole from KDE 3.x.

FYI the "workaround" I use here for the moment (if I don't just disable the 
feature with PATCHRESOLVE = "noop") is to use xterm instead of Konsole for 
TERMCMD and TERMCMDRUN.

Cheers,
Paul

[1] http://lists.kde.org/?l=kde-core-devel&m=123188146810950&w=2