[Buildroot] [Help] Building from a private PyPi server

[Buildroot] [Help] Building from a private PyPi server

[Alberto Fahrenkrog]

[-- Attachment #1.1.1: Type: text/plain, Size: 1483 bytes --]

Hello all,

In our company we have custom Python packages stored in a private PyPi server. To install them we simply add the --extra-index-url in our requirements.txt file when using pip. The packages are also available for manual download from the server, however all URLs to any .tar.gz file are in the form https://git.ourserver.com/packages/pypi/package-name/version-number/files/1234. This leads me to the following options (I think):


  *   Try to get Buildroot to use the private PyPi server. I had a look at the "scanpypi" script and it generated the .mk and .in files for a regular PyPi package, but I could not find a way to tell scanpypi to use an extra url or private pypi address
  *   Find a way, using wget, to download the package from the address above. Unfortunately I've had no luck with that yet. With the command: wget https://ourserver.com/-/packages/pypi/package-name/version_nr/files/5525 --header 'Authorization: token AUTH_TOKEN' --output-document=filename.tar.gz I was not able to get a correct tar.gz file. But that's more my problem than a Buildroot issue.
  *   Download the file manually and have Buildroot use a local file. I will try that for the time being, although it seems the least automated possibility.

If you guys have any other ideas I'd like to hear them.

Cheers,

Alberto

Alberto Fahrenkrog Senior Embedded Engineer
Orica - Perth Technology Centre
p: +61 (0) 8 6365 4056
e: Alberto.Fahrenkrog@rigtechnologies.com.au

[-- Attachment #1.1.2: Type: text/html, Size: 6055 bytes --]

[-- Attachment #1.2: image497442.png --]
[-- Type: image/png, Size: 14270 bytes --]

[-- Attachment #2: Type: text/plain, Size: 150 bytes --]

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [Help] Building from a private PyPi server

[Alberto Fahrenkrog]

[-- Attachment #1.1.1: Type: text/plain, Size: 2360 bytes --]

Apologies,

Forgot to mention that I can also get a .whl file of the python package manually. Can Buildroot work with a .whl file?

Cheers,

Alberto

Alberto Fahrenkrog Senior Embedded Engineer
Orica - Perth Technology Centre
p: +61 (0) 8 6365 4056
e: Alberto.Fahrenkrog@rigtechnologies.com.au
________________________________
From: buildroot <buildroot-bounces@buildroot.org> on behalf of Alberto Fahrenkrog <Alberto.Fahrenkrog@rigtechnologies.com.au>
Sent: Monday, February 27, 2023 8:17 AM
To: buildroot@buildroot.org <buildroot@buildroot.org>
Subject: [Buildroot] [Help] Building from a private PyPi server

You don't often get email from alberto.fahrenkrog@rigtechnologies.com.au. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification>
Hello all,

In our company we have custom Python packages stored in a private PyPi server. To install them we simply add the --extra-index-url in our requirements.txt file when using pip. The packages are also available for manual download from the server, however all URLs to any .tar.gz file are in the form https://git.ourserver.com/packages/pypi/package-name/version-number/files/1234. This leads me to the following options (I think):


  *   Try to get Buildroot to use the private PyPi server. I had a look at the "scanpypi" script and it generated the .mk and .in files for a regular PyPi package, but I could not find a way to tell scanpypi to use an extra url or private pypi address
  *   Find a way, using wget, to download the package from the address above. Unfortunately I've had no luck with that yet. With the command: wget https://ourserver.com/-/packages/pypi/package-name/version_nr/files/5525 --header 'Authorization: token AUTH_TOKEN' --output-document=filename.tar.gz I was not able to get a correct tar.gz file. But that's more my problem than a Buildroot issue.
  *   Download the file manually and have Buildroot use a local file. I will try that for the time being, although it seems the least automated possibility.

If you guys have any other ideas I'd like to hear them.

Cheers,

Alberto
[cid:image497442.png@9FC6413F.5C1BC59F]
Alberto​ Fahrenkrog

Senior Embedded Engineer
Orica ‑ Perth Technology Centre
p: +61 (0) 8 6365 4056
e: Alberto.Fahrenkrog@rigtechnologies.com.au<mailto:Alberto.Fahrenkrog@rigtechnologies.com.au>

[-- Attachment #1.1.2: Type: text/html, Size: 12948 bytes --]

[-- Attachment #1.2: image497442.png --]
[-- Type: image/png, Size: 14270 bytes --]

[-- Attachment #1.3: image695658.png --]
[-- Type: image/png, Size: 14270 bytes --]

[-- Attachment #2: Type: text/plain, Size: 150 bytes --]

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [Help] Building from a private PyPi server

[Peter Korsgaard]

>>>>> "Alberto" == Alberto Fahrenkrog <Alberto.Fahrenkrog@rigtechnologies.com.au> writes:

 > Hello all,
 > In our company we have custom Python packages stored in a private PyPi
 > server. To install them we simply add the --extra-index-url in our
 > requirements.txt file when using pip.

Are you aware of the security issues with --extra-index-url?

https://github.com/pypa/pip/issues/9612

 > The packages are also available for manual download from the server,
 > however all URLs to any .tar.gz file are in the form
 > https://git.ourserver.com/packages/pypi/package-name/version-number/files/1234. This
 > leads me to the following options (I think):


 >   * Try to get Buildroot to use the private PyPi server. I had a look
 > at the "scanpypi" script and it generated the .mk and .in files for a
 > regular PyPi package, but I could not find a way to tell scanpypi to
 > use an extra url or private pypi address

scanpypi is indeed hard coded to use pypi and their JSON API
(https://pypi.org/pypi/{pkg}/json), so not trivial to use with a local
repo. Do your private server expose the same API? If so, I guess we
could add an argument to scanpypi to use a custom URI instead of
pypi.org.


 >   * Find a way, using wget, to download the package from the address
 > above. Unfortunately I've had no luck with that yet. With the command:
 > wget
 > https://ourserver.com/-/packages/pypi/package-name/version_nr/files/5525

Notice: You have an additional /-/ compared to the URL above.

-- 
Bye, Peter Korsgaard
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [Help] Building from a private PyPi server

[Alberto Fahrenkrog]

[-- Attachment #1.1.1: Type: text/plain, Size: 2728 bytes --]

Hi Peter,

I believe the API used (JSON) is the same (similar?), I can confirm with one of my colleagues. Is scanpypi written in C or Python? I could have a look at the code. I don't know where the source is, but I'll look it up.

The typo was accidental (below), the fact is that even manually using wget I have not yet figured out how to read from the link, also something I will follow up with my colleague.

I wasn't aware of the --extra-index-url, I'll pass that information along to our software guys.

I found a workaround to get it to work, using our repos directly. I need to sort some things out with our software department, but I'm making progress in the right direction.

Cheers,

Alberto



Alberto Fahrenkrog Senior Embedded Engineer
Orica - Perth Technology Centre
p: +61 (0) 8 6365 4056
e: Alberto.Fahrenkrog@rigtechnologies.com.au
________________________________
From: Peter Korsgaard <peter@korsgaard.com>
Sent: Monday, February 27, 2023 10:57 PM
To: Alberto Fahrenkrog <Alberto.Fahrenkrog@rigtechnologies.com.au>
Cc: buildroot@buildroot.org <buildroot@buildroot.org>
Subject: Re: [Help] Building from a private PyPi server

>>>>> "Alberto" == Alberto Fahrenkrog <Alberto.Fahrenkrog@rigtechnologies.com.au> writes:

 > Hello all,
 > In our company we have custom Python packages stored in a private PyPi
 > server. To install them we simply add the --extra-index-url in our
 > requirements.txt file when using pip.

Are you aware of the security issues with --extra-index-url?

https://github.com/pypa/pip/issues/9612

 > The packages are also available for manual download from the server,
 > however all URLs to any .tar.gz file are in the form
 > https://git.ourserver.com/packages/pypi/package-name/version-number/files/1234. This
 > leads me to the following options (I think):


 >   * Try to get Buildroot to use the private PyPi server. I had a look
 > at the "scanpypi" script and it generated the .mk and .in files for a
 > regular PyPi package, but I could not find a way to tell scanpypi to
 > use an extra url or private pypi address

scanpypi is indeed hard coded to use pypi and their JSON API
(https://pypi.org/pypi/{pkg}/json), so not trivial to use with a local
repo. Do your private server expose the same API? If so, I guess we
could add an argument to scanpypi to use a custom URI instead of
pypi.org.


 >   * Find a way, using wget, to download the package from the address
 > above. Unfortunately I've had no luck with that yet. With the command:
 > wget
 > https://ourserver.com/-/packages/pypi/package-name/version_nr/files/5525

Notice: You have an additional /-/ compared to the URL above.

--
Bye, Peter Korsgaard

[-- Attachment #1.1.2: Type: text/html, Size: 9739 bytes --]

[-- Attachment #1.2: image520031.png --]
[-- Type: image/png, Size: 14270 bytes --]

[-- Attachment #2: Type: text/plain, Size: 150 bytes --]

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [Help] Building from a private PyPi server

[Peter Korsgaard]

>>>>> "Alberto" == Alberto Fahrenkrog <Alberto.Fahrenkrog@rigtechnologies.com.au> writes:

 > Hi Peter,
 > I believe the API used (JSON) is the same (similar?), I can confirm
 > with one of my colleagues. Is scanpypi written in C or Python? I could
 > have a look at the code. I don't know where the source is, but I'll
 > look it up.

It is a python script, just have a look at utils/scanpypi.


 > The typo was accidental (below), the fact is that even manually using
 > wget I have not yet figured out how to read from the link, also
 > something I will follow up with my colleague.

 > I wasn't aware of the --extra-index-url, I'll pass that information
 > along to our software guys.

 > I found a workaround to get it to work, using our repos directly. I
 > need to sort some things out with our software department, but I'm
 > making progress in the right direction.

OK.

-- 
Bye, Peter Korsgaard
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot