Why do we use /usr/lib/python to install python libraries?

Why do we use /usr/lib/python to install python libraries?

[Anthony Liguori]

This seems exceptionally evil and is definitely a PITA.  What was the 
original problem that led us to do this?  I'd like to take a stab at 
correcting it so we can have a more sane installation.

Regards,

Anthony Liguori

Re: Why do we use /usr/lib/python to install python libraries?

[Mark Williamson]

> This seems exceptionally evil and is definitely a PITA.  What was the
> original problem that led us to do this?  I'd like to take a stab at
> correcting it so we can have a more sane installation.

I thought it involved some pain early one with supporting different Python 
versions in different distros?  Maybe with the binary installer?

Of course I could be talking bunk - it's been like that for a loooooong time 
and my memory was better when I was younger ;-)

Cheers,
Mark

-- 
Dave: Just a question. What use is a unicyle with no seat?  And no pedals!
Mark: To answer a question with a question: What use is a skateboard?
Dave: Skateboards have wheels.
Mark: My wheel has a wheel!

Re: Why do we use /usr/lib/python to install python libraries?

[Anthony Liguori]

Mark Williamson wrote:
>> This seems exceptionally evil and is definitely a PITA.  What was the
>> original problem that led us to do this?  I'd like to take a stab at
>> correcting it so we can have a more sane installation.
>>     
>
> I thought it involved some pain early one with supporting different Python 
> versions in different distros?  Maybe with the binary installer?
>
> Of course I could be talking bunk - it's been like that for a loooooong time 
> and my memory was better when I was younger ;-)
>   

Yeah, this is why I asked.  I have vague recollections of this 
discussion too but can't remember the exact reasons.  I'm hoping that 
we've gotten smarter since then and can find a better solution :-)

Regards,

Anthony Liguori

> Cheers,
> Mark
>
>   

Re: Why do we use /usr/lib/python to install python libraries?

[Keir Fraser]

On 9 Jun 2006, at 03:00, Anthony Liguori wrote:

> Mark Williamson wrote:
>>> This seems exceptionally evil and is definitely a PITA.  What was the
>>> original problem that led us to do this?  I'd like to take a stab at
>>> correcting it so we can have a more sane installation.
>>>
>>
>> I thought it involved some pain early one with supporting different 
>> Python versions in different distros?  Maybe with the binary 
>> installer?
>>
>> Of course I could be talking bunk - it's been like that for a 
>> loooooong time and my memory was better when I was younger ;-)
>>
>
> Yeah, this is why I asked.  I have vague recollections of this 
> discussion too but can't remember the exact reasons.  I'm hoping that 
> we've gotten smarter since then and can find a better solution :-)

To be able to install on a machine different from the one we built on. 
Because our distribution method is to 'python install' into a dist/ 
subdirectory and then basically blat that directory into / on the 
target machine. It would probably be smarter to include python scripts 
to be able to 'python install' out of dist/ -- it'd require us to have 
python-devel and C compiler installed on the target box though.

Ultimately the argument was that distros should install in their usual 
place because they *know* what python version they ship, but that since 
we didn't we'd just install in /usr/lib/python because that required 
least thought for maximum coverage. :-)

  -- Keir

Re: Why do we use /usr/lib/python to install python libraries?

[Anthony Liguori]

Keir Fraser wrote:
>
> On 9 Jun 2006, at 03:00, Anthony Liguori wrote:
>
>> Mark Williamson wrote:
>>>> This seems exceptionally evil and is definitely a PITA.  What was the
>>>> original problem that led us to do this?  I'd like to take a stab at
>>>> correcting it so we can have a more sane installation.
>>>>
>>>
>>> I thought it involved some pain early one with supporting different 
>>> Python versions in different distros?  Maybe with the binary installer?
>>>
>>> Of course I could be talking bunk - it's been like that for a 
>>> loooooong time and my memory was better when I was younger ;-)
>>>
>>
>> Yeah, this is why I asked.  I have vague recollections of this 
>> discussion too but can't remember the exact reasons.  I'm hoping that 
>> we've gotten smarter since then and can find a better solution :-)
>
> To be able to install on a machine different from the one we built on. 
> Because our distribution method is to 'python install' into a dist/ 
> subdirectory and then basically blat that directory into / on the 
> target machine. It would probably be smarter to include python scripts 
> to be able to 'python install' out of dist/ -- it'd require us to have 
> python-devel and C compiler installed on the target box though.

Does anyone make use of the whole dist/ directory separation?  I've 
always found that to be very odd.

Regards,

Anthony Liguori

>
> Ultimately the argument was that distros should install in their usual 
> place because they *know* what python version they ship, but that 
> since we didn't we'd just install in /usr/lib/python because that 
> required least thought for maximum coverage. :-)
>
>  -- Keir
>

Re: Why do we use /usr/lib/python to install python libraries?

[Matt Ayres]

Anthony Liguori wrote:
> Keir Fraser wrote:
>>
>> On 9 Jun 2006, at 03:00, Anthony Liguori wrote:
>>
>>> Mark Williamson wrote:
>>>>> This seems exceptionally evil and is definitely a PITA.  What was the
>>>>> original problem that led us to do this?  I'd like to take a stab at
>>>>> correcting it so we can have a more sane installation.
>>>>>
>>>>
>>>> I thought it involved some pain early one with supporting different 
>>>> Python versions in different distros?  Maybe with the binary installer?
>>>>
>>>> Of course I could be talking bunk - it's been like that for a 
>>>> loooooong time and my memory was better when I was younger ;-)
>>>>
>>>
>>> Yeah, this is why I asked.  I have vague recollections of this 
>>> discussion too but can't remember the exact reasons.  I'm hoping that 
>>> we've gotten smarter since then and can find a better solution :-)
>>
>> To be able to install on a machine different from the one we built on. 
>> Because our distribution method is to 'python install' into a dist/ 
>> subdirectory and then basically blat that directory into / on the 
>> target machine. It would probably be smarter to include python scripts 
>> to be able to 'python install' out of dist/ -- it'd require us to have 
>> python-devel and C compiler installed on the target box though.
> 
> Does anyone make use of the whole dist/ directory separation?  I've 
> always found that to be very odd.
> 

/me waves.. I never use make install, only make dist. Then I use the 
dist directory to maintain updates across hosts.  The other main reason 
is because I never want to actually install Xen when I build it.

Re: Why do we use /usr/lib/python to install python libraries?

[Ewan Mellor]

On Thu, Jun 08, 2006 at 08:09:06PM -0500, Anthony Liguori wrote:

> This seems exceptionally evil and is definitely a PITA.  What was the 
> original problem that led us to do this?  I'd like to take a stab at 
> correcting it so we can have a more sane installation.

If you're going to fix this, then great.  Perhaps you could make it:

a) use the proper Python distutils if installed;

b) fall back to using /usr/lib{,64}/python if distutils is not installed
   (though I can't imagine that there are many systems that are shipping with
   Python but without distutils, and you do need Python to run Xend, after
   all);

c) make sure that we actually do install into /usr/lib64 on appropriate
   systems.  This is a current bug, though I've no idea when it got
   introduced.  We need to install into /usr/lib64 because xc.so and xs.so are
   compiled native code.

Thanks Anthony,

Ewan.

Re: Why do we use /usr/lib/python to install python libraries?

[Anthony Liguori]

Ewan Mellor wrote:
> On Thu, Jun 08, 2006 at 08:09:06PM -0500, Anthony Liguori wrote:
>
>   
>> This seems exceptionally evil and is definitely a PITA.  What was the 
>> original problem that led us to do this?  I'd like to take a stab at 
>> correcting it so we can have a more sane installation.
>>     
>
> If you're going to fix this, then great.  Perhaps you could make it:
>   

I would love to, if patches are willing to be taken.

> a) use the proper Python distutils if installed;
>   

I think we already are aren't we?  I thought the crux of the problem was 
that people are relying on having a deterministic install path so that 
it can be deployed on many machines (regardless of Python version).  We 
would have to break this "feature".  I'm happy to do that of course :-)

Regards,

Anthony Liguori

> b) fall back to using /usr/lib{,64}/python if distutils is not installed
>    (though I can't imagine that there are many systems that are shipping with
>    Python but without distutils, and you do need Python to run Xend, after
>    all);
>
> c) make sure that we actually do install into /usr/lib64 on appropriate
>    systems.  This is a current bug, though I've no idea when it got
>    introduced.  We need to install into /usr/lib64 because xc.so and xs.so are
>    compiled native code.
>
> Thanks Anthony,
>
> Ewan.
>   

Re: Why do we use /usr/lib/python to install python libraries?

[Ewan Mellor]

On Wed, Jun 14, 2006 at 06:45:12AM -0500, Anthony Liguori wrote:

> Ewan Mellor wrote:
> >On Thu, Jun 08, 2006 at 08:09:06PM -0500, Anthony Liguori wrote:
> >
> >  
> >>This seems exceptionally evil and is definitely a PITA.  What was the 
> >>original problem that led us to do this?  I'd like to take a stab at 
> >>correcting it so we can have a more sane installation.
> >>    
> >
> >If you're going to fix this, then great.  Perhaps you could make it:
> >  
> 
> I would love to, if patches are willing to be taken.

Certainly we'll take the patch.

> >a) use the proper Python distutils if installed;
> >  
> 
> I think we already are aren't we?  I thought the crux of the problem was 
> that people are relying on having a deterministic install path so that 
> it can be deployed on many machines (regardless of Python version).  We 
> would have to break this "feature".  I'm happy to do that of course :-)

I meant use distutils in the install script, i.e. on the destination machine.
What people want to be able to do is create a dist directory, copy it
wholesale over to a target host, and then run install.sh and have the right
thing happen.  If the install.sh was clever enough to drop things into the
correct Python directory, then that's great and doesn't break anything.

In the case where people use "make install" directly, yes, distutils is used
already.

Some people copy the dist directory straight onto the root directory on their
test box (many of the developers here, for example).  That's not really
supported, because you end up with both the hotplug and udev scripts
installed, for example, but it is handy for developers, so it would be good to
preserve that behaviour.  In other words, you still need a
dist/usr/lib/python/ directory to be created, even if install.sh doesn't copy
those files directly to that directory when it runs.

Thanks,

Ewan.

Re: Why do we use /usr/lib/python to install python libraries?

[John Levon]

On Wed, Jun 14, 2006 at 08:32:39AM +0100, Ewan Mellor wrote:

> c) make sure that we actually do install into /usr/lib64 on appropriate
>    systems.  This is a current bug, though I've no idea when it got
>    introduced.  We need to install into /usr/lib64 because xc.so and xs.so are
>    compiled native code.

Could you clarify this? What happens on 64-bit Linux installations right
now?

Since the 64-bit cleanness isn't quite there yet, we need to install
both a 64-bit and 32-bit on Solaris. Unfortunately Python doesn't seem
to have much support for running like this (in particular, it'll just
give up if it finds an extension .so of the wrong elf class rather than
looking in a 64-bit path or whatever).

thanks
john

Re: Why do we use /usr/lib/python to install python libraries?

[Ewan Mellor]

On Wed, Jun 14, 2006 at 04:31:56PM +0100, John Levon wrote:

> On Wed, Jun 14, 2006 at 08:32:39AM +0100, Ewan Mellor wrote:
> 
> > c) make sure that we actually do install into /usr/lib64 on appropriate
> >    systems.  This is a current bug, though I've no idea when it got
> >    introduced.  We need to install into /usr/lib64 because xc.so and xs.so are
> >    compiled native code.
> 
> Could you clarify this? What happens on 64-bit Linux installations right
> now?
> 
> Since the 64-bit cleanness isn't quite there yet, we need to install
> both a 64-bit and 32-bit on Solaris. Unfortunately Python doesn't seem
> to have much support for running like this (in particular, it'll just
> give up if it finds an extension .so of the wrong elf class rather than
> looking in a 64-bit path or whatever).

My belief was that, because we have native code, on those platforms that
normally use /usr/lib64, we ought to be putting our stuff in /usr/lib64.
Does anyone know whether this is the case?

Currently, all the Python stuff goes into /usr/lib/python, even on platforms
with a /usr/lib64 directory.  I was assuming that that was a bug, but I don't
know for sure.

Ewan.