Re: [PATCH 3/7] bb: add bb.fetcher as dynamic fetch module instance for fetch/fetch2

[Joshua Lock <josh@openedhand.com>]

On Mon, 2011-01-10 at 20:19 +0000, Richard Purdie wrote:
> On Mon, 2011-01-10 at 16:43 +0000, Richard Purdie wrote:
> > On Mon, 2011-01-10 at 11:42 +0000, Richard Purdie wrote:
> > > On Mon, 2011-01-10 at 10:13 +0000, Joshua Lock wrote:
> > > > On Sat, 2011-01-08 at 15:15 +0800, Yu Ke wrote:
> > > > > +import bb
> > > > > +from bb import fetch, fetch2
> > > > > +
> > > > > +# switch between fetch and fetch2
> > > > > +instance = bb.fetch
> > > > > +__version__ = "1"
> > > > > +
> > > > > +#instance = bb.fetch2
> > > > > +#__version__ = "2"
> > > > 
> > > > I'm not very happy with this, it doesn't feel right. Can we not add some
> > > > runtime determination of which module to use (a variable in
> > > > bitbake.conf?) and then use some dynamic Python magic to add it to the
> > > > imported modules list?
> > > > 
> > > > A quick Google makes me thing we could do something more dynamic with
> > > > the __import__() function.
> > > 
> > > Agreed, I was also thinking about this but wanted to come up with a
> > > suggestion about how to better handle it. There is some interesting info
> > > in this webpage:
> > > 
> > > http://stackoverflow.com/questions/1096216/override-namespace-in-python
> > > 
> > > This example in particular looks useful:
> > > 
> > > def weirdimport(fullpath):
> > >   global project
> > > 
> > >   import os
> > >   import sys
> > >   sys.path.append(os.path.dirname(fullpath))
> > >   try:
> > >       project = __import__(os.path.basename(fullpath))
> > >       sys.modules['project'] = project
> > >   finally:
> > >       del sys.path[-1]
> > > 
> > > Although we shouldn't need to manipulate sys.path in our case.
> > > 
> > > As Joshua mentions, overriding the __import__ builtin should also work
> > > and might avoid some ordering issues we'd have with the above.
> > 
> > I did come up with this code inserted at the start of bin/bitbake:
> > 
> > origimport = __builtins__.__import__
> > def bbimport(name, *args, **kwargs):
> >     newname = name
> >     if name.find("bb.fetch") != -1:
> >         newname = name.replace("bb.fetch", "bb.fetch2")
> >     ret = origimport(newname, *args, **kwargs)
> >     if name != newname:
> >         sys.modules[name] = sys.modules[newname]
> >         if name == "bb.fetch":
> >             bb.fetch = sys.modules[newname]
> >     return ret
> > if os.environ.get('BBFETCH2'):
> >     __builtins__.__import__ = bbimport
> > 
> > which is pretty ugly :(. The big advantage of this is that it does
> > redirect any bb.fetch calls into bb.fetch2 calls, even within the
> > bb.fetch2 codebase.
> > 
> > I'm working through some other ideas as time permits.
> 
> If we assume bb.fetch2 will not make calls to bb.fetch we can simplify
> this to:
> 
> if os.environ.get("BBFETCH2"):
>     from bb import fetch2 as fetch
>     sys.modules['bb.fetch'] = sys.modules['bb.fetch2']

Lovely, this is almost exactly what I had envisaged :-)

Cheers,
Joshua

> 
> so from the environment you can trigger the new fetcher code. For Poky,
> I've added something to hardcode that behaviour since I don't want it to
> break.
> 
> I've pushed this both into Poky and bitbake upstream so we're in sync
> and can move forward on improving the fetcher code :).
> 
> Cheers,
> 
> Richard
> 
> 
> 
> 
> 
> 

-- 
Joshua Lock
        Intel Open Source Technology Centre