From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by mx.groups.io with SMTP id smtpd.web12.9185.1585956143036486143 for ; Fri, 03 Apr 2020 16:22:23 -0700 Authentication-Results: mx.groups.io; dkim=missing; spf=none, err=permanent DNS error (domain: linux.intel.com, ip: 192.55.52.151, mailfrom: timothy.t.orling@linux.intel.com) IronPort-SDR: Gez9speEnwOsmTPNzGO9PvDh4LvR/A/+8vi6KYFWFfHNGROLEUmto+OQXiStO3XpHdEeQjAAnN N19bsvy6b6pQ== X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Apr 2020 16:22:22 -0700 IronPort-SDR: QGZhcNy0ket72lZzYwyr3oOgztAe6098UpZfW+iUgR3SCqPB0Ec+AtCOnRITmMvxddakrRz3Gb ilRQIDQgo+mQ== X-IronPort-AV: E=Sophos;i="5.72,341,1580803200"; d="scan'208,217";a="250310974" Received: from amspicke-mobl.amr.corp.intel.com ([10.255.230.64]) by orsmga003-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Apr 2020 16:22:21 -0700 From: "Tim Orling" Message-Id: Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.80.23.2.2\)) Subject: Re: [OE-core] [PATCH 1/4] scripts/install-buildtools: improvements Date: Fri, 3 Apr 2020 16:22:20 -0700 In-Reply-To: <9bd493e6c0a4117b04e636bc0f3200e84ab30cbd.camel@linuxfoundation.org> Cc: Tim Orling , OE Core mailing list To: Richard Purdie , Andre McCurdy References: <20200330194302.30032-1-timothy.t.orling@linux.intel.com> <1601CF873C42565A.6346@lists.openembedded.org> <9bd493e6c0a4117b04e636bc0f3200e84ab30cbd.camel@linuxfoundation.org> X-Mailer: Apple Mail (2.3608.80.23.2.2) Content-Type: multipart/alternative; boundary="Apple-Mail=_54294D89-DAFE-4E48-9885-88A9D1E6DEEC" --Apple-Mail=_54294D89-DAFE-4E48-9885-88A9D1E6DEEC Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On Apr 1, 2020, at 3:16 PM, Richard Purdie wrote: >=20 > On Wed, 2020-04-01 at 15:09 -0700, Andre McCurdy wrote: >> On Wed, Apr 1, 2020 at 3:02 PM Tim Orling wrote: >>> On Wed, Apr 1, 2020 at 2:32 PM Tim Orling via >>> lists.openembedded.org >>> wrote: >>>> On Wed, Apr 1, 2020 at 1:09 PM Andre McCurdy >>>> wrote: >>>>> It looks like there are some assumptions about the minimum >>>>> version of >>>>> python3 on the host in order to even be able to run >>>>> install-buildtools. Should that be checked explicitly? >>>>=20 >>>> To be honest, everything was written on CentOS-7, where =E2=80=98pyth= on3=E2=80=99 >>>> is now python36. >>>>=20 >>>> The remaining supported holdout for 3.4 is Debian-8 (Jessie). >>>> Which also happens to have tar 1.27. At least it has git 2.1.4, >>>> so it=E2=80=99s not a complete mess. >>>>=20 >>>> Python 3.4 is already EOL. We should not support it for the >>>> complete build system, but I can see that the install-buildtools >>>> script should ideally still run on python34. The whole point is >>>> to make it easier to install the pre-built buildtools. >>>=20 >>> Python 3.5 is now required in bitbake/lib/bb/__init__.py >>> Which is needed for bb.utils.md5sum_file and sha256_file to check >>> the checksum of the download. I=E2=80=99d rather not have to rewrite t= hose >>> functions. >>=20 >> Looks like copy and pasting ~30 lines? Maybe not even much of a net >> increase in code size if you can remove the existing code to check >> that bitbake libs are available... >>=20 >> def _hasher(method, filename): >> import mmap >>=20 >> with open(filename, "rb") as f: >> try: >> with mmap.mmap(f.fileno(), 0, access=3Dmmap.ACCESS_READ) as >> mm: >> for chunk in iter(lambda: mm.read(8192), b''): >> method.update(chunk) >> except ValueError: >> # You can't mmap() an empty file so silence this >> exception >> pass >> return method.hexdigest() >>=20 >> def md5_file(filename): >> """ >> Return the hex string representation of the MD5 checksum of >> filename. >> """ >> import hashlib >> return _hasher(hashlib.md5(), filename) >>=20 >> def sha256_file(filename): >> """ >> Return the hex string representation of the 256-bit SHA checksum >> of >> filename. >> """ >> import hashlib >> return _hasher(hashlib.sha256(), filename) >=20 > Can you send a patch against master-next please? The buildtools > installer is one place we probably do want to support older pythons and > it shouldn't be too onnerous. >=20 Submitted. The check for whether the install worked or not was the hardest= part. After many iterations, I just could not get subprocess.Popen to work as ex= pected on python 3.4. https://patchwork.openembedded.org/patch/171593/ Thank you Andre for the testing and input. It takes a village to make a go= od release. > Cheers, >=20 > Richard >=20 >=20 --Apple-Mail=_54294D89-DAFE-4E48-9885-88A9D1E6DEEC Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8

On Apr 1, 20= 20, at 3:16 PM, Richard Purdie <richard.purdie@linuxfoundation.org> wrote:<= /div>
On Wed, 2020-04-01 at 15:09 -0700, Andre McCurdy wrote:
On Wed, Apr 1, 2020 at 3:02 PM Tim Orling= <ticotimo@gmail.com> wrote:
On Wed, Apr= 1, 2020 at 2:32 PM Tim Orling via
lists.openembedded.org <ticotimo=3Dgmail.c= om@lists.openembedded.org>
wrote:
On Wed, Apr 1, 2020 at 1:09 PM Andre McCurdy= <armccurdy@gmail.com<= /a>
wrote:
It looks like there are some assumptions about the minimum
v= ersion of
python3 on the host in order to even be able to run=
install-buildtools. Should that be checked explicitly?

To be honest, everything was written o= n CentOS-7, where =E2=80=98python3=E2=80=99
is now python36.<= br class=3D"">
The remaining supported holdout for 3.4 is Deb= ian-8 (Jessie).
Which also happens to have tar 1.27. At least= it has git 2.1.4,
so it=E2=80=99s not a complete mess.

Python 3.4 is already EOL. We should not support it= for the
complete build system, but I can see that the instal= l-buildtools
script should ideally still run on python34. The= whole point is
to make it easier to install the pre-built bu= ildtools.

Python 3.5 is now requi= red in bitbake/lib/bb/__init__.py
Which is needed for bb.util= s.md5sum_file and sha256_file to check
the checksum of the do= wnload. I=E2=80=99d rather not have to rewrite those
function= s.

Looks like copy and pasting ~3= 0 lines? Maybe not even much of a net
increase in code size i= f you can remove the existing code to check
that bitbake libs= are available...

def _hasher(method, filename= ):
   import mmap

   with open(filename, "rb") as f:
 &= nbsp;     try:
   &n= bsp;       with mmap.mmap(f.fileno(), 0,= access=3Dmmap.ACCESS_READ) as
mm:
  = ;            &n= bsp;for chunk in iter(lambda: mm.read(8192), b''):
 &nb= sp;            =      method.update(chunk)
 &nb= sp;     except ValueError:
 &n= bsp;         # You can't mmap(= ) an empty file so silence this
exception
&nbs= p;          pass
   return method.hexdigest()

def md5_file(filename):
   """
   Return the hex string representation of the MD5 c= hecksum of
filename.
   """
   import hashlib
  &nb= sp;return _hasher(hashlib.md5(), filename)

def= sha256_file(filename):
   """
=    Return the hex string representation of the 256-bit SHA = checksum
of
   filename.
   """
   import hash= lib
   return _hasher(hashlib.sha256(), filen= ame)

Can you send a patch against= master-next please? The buildtools
installer is one place we= probably do want to support older pythons and
it shouldn't b= e too onnerous.

https://p= atchwork.openembedded.org/patch/171593/

=
Thank you Andre for the testing and input. It takes a village to make = a good release.

Cheers,

Ric= hard


<= /div>
--Apple-Mail=_54294D89-DAFE-4E48-9885-88A9D1E6DEEC--