* Re: [meta-networking][PATCH] GeoIP: change data fetching strategy
2015-01-08 3:41 ` Khem Raj
@ 2015-01-08 11:14 ` Martin Jansa
2015-01-09 17:02 ` Joe MacDonald
1 sibling, 0 replies; 4+ messages in thread
From: Martin Jansa @ 2015-01-08 11:14 UTC (permalink / raw)
To: openembedded-devel
[-- Attachment #1: Type: text/plain, Size: 5484 bytes --]
On Wed, Jan 07, 2015 at 07:41:06PM -0800, Khem Raj wrote:
>
> > On Jan 7, 2015, at 6:19 PM, Joe MacDonald <joe_macdonald@mentor.com> wrote:
> >
> > Since the GeoIP databases are not versioned and they essentially change
> > without notice, we cannot reasonably expect the checksums to be consistent
>
> isn’t this like AUTOREV ?. IMO such packages should be mirrored somewhere
> regularly and used. if you maintain this recipe then please mirror the sources
> somewhere and then update them regularly, but from one build to another is just
> too frequent.
Agreed, we should also make sure that once PREMIRROR is fully populated
we can build without any access to network. This is breaking that
assumption and doesn't even use PREMIRROR to "cache" them.
> > from one build to the next. Therefore, don't download them using the
> > usual wget fetcher, but do it manually at the end of the do_fetch stage
> > and place the uncompressed data files where the remainder of the build
> > expects they would've been placed.
> >
> > Signed-off-by: Joe MacDonald <joe_macdonald@mentor.com>
> > ---
> >
> > I considered trying to use the wget fetcher and having it skip over the
> > checksum operation and, honestly, I'd still prefer to do it that way, but
> > looking around and at the fetch2 code, it doesn't appear to be able to support
> > that outside of a BB_STRICT_CHECKSUM="" setting, which we definitely don't want
> > to do here.
> >
> > -J.
> >
> > .../recipes-support/geoip/geoip_1.6.0.bb | 36 +++++++++-------------
> > 1 file changed, 14 insertions(+), 22 deletions(-)
> >
> > diff --git a/meta-networking/recipes-support/geoip/geoip_1.6.0.bb b/meta-networking/recipes-support/geoip/geoip_1.6.0.bb
> > index 65e8362..71fda3b 100644
> > --- a/meta-networking/recipes-support/geoip/geoip_1.6.0.bb
> > +++ b/meta-networking/recipes-support/geoip/geoip_1.6.0.bb
> > @@ -8,28 +8,9 @@ using reverse DNS lookups."
> > HOMEPAGE = "http://dev.maxmind.com/geoip/"
> > SECTION = "Development/Libraries"
> >
> > -SRC_URI = "http://www.maxmind.com/download/geoip/api/c/GeoIP-1.6.0.tar.gz;name=tarball \
> > - http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz;apply=no;name=GeoIP-dat \
> > - http://geolite.maxmind.com/download/geoip/database/GeoIPv6.dat.gz;apply=no;name=GeoIPv6-dat \
> > - http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz;apply=no;name=GeoLiteCity-dat \
> > - http://geolite.maxmind.com/download/geoip/database/GeoLiteCityv6-beta/GeoLiteCityv6.dat.gz;apply=no;name=GeoLiteCityv6-dat \
> > -"
> > -
> > -SRC_URI[tarball.md5sum] = "89f4cdfdab43f1d67364cd7c85bbe8ca"
> > -SRC_URI[tarball.sha256sum] = "075a0c2815cd099e9ec35c9569db716a3fefcdbb6a10dbfa1ce7c6cd48d4a635"
> > -
> > -SRC_URI[GeoIP-dat.md5sum] = "4bc1e8280fe2db0adc3fe48663b8926e"
> > -SRC_URI[GeoIP-dat.sha256sum] = "7fd7e4829aaaae2677a7975eeecd170134195e5b7e6fc7d30bf3caf34db41bcd"
> > -
> > -SRC_URI[GeoIPv6-dat.md5sum] = "aac7e6e9b141de80934ecee52daf7f56"
> > -SRC_URI[GeoIPv6-dat.sha256sum] = "126fd2953eb193e60538e30b4465610530383f7a782745cacdca5ba6825f471c"
> > -
> > -SRC_URI[GeoLiteCity-dat.md5sum] = "15a42c684c53d2309e6632a6d6e02531"
> > -SRC_URI[GeoLiteCity-dat.sha256sum] = "5ec02a6d39d545c77ec12cc30c6a8856883d8f55522fc5cd4f25af80163c6b3c"
> > -
> > -SRC_URI[GeoLiteCityv6-dat.md5sum] = "49d6ec946fa0a2575b5112a68d71f933"
> > -SRC_URI[GeoLiteCityv6-dat.sha256sum] = "7a345e6cf0e59f8ab589ff15020241f0b03342dd04cc584f814c4f4700d49405"
> > -
> > +SRC_URI = "http://www.maxmind.com/download/geoip/api/c/GeoIP-1.6.0.tar.gz"
> > +SRC_URI[md5sum] = "89f4cdfdab43f1d67364cd7c85bbe8ca"
> > +SRC_URI[sha256sum] = "075a0c2815cd099e9ec35c9569db716a3fefcdbb6a10dbfa1ce7c6cd48d4a635"
> >
> > LICENSE = "LGPL-2.1"
> >
> > @@ -43,6 +24,17 @@ inherit autotools
> > EXTRA_OECONF = "--disable-static \
> > --disable-dependency-tracking "
> >
> > +do_fetch_append() {
> > + os.system("${FETCHCMD_wget} http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz");
> > + os.system("gunzip -c ${DL_DIR}/GeoIP.dat.gz > ${WORKDIR}/GeoIP.dat");
> > + os.system("${FETCHCMD_wget} http://geolite.maxmind.com/download/geoip/database/GeoIPv6.dat.gz");
> > + os.system("gunzip -c ${DL_DIR}/GeoIPv6.dat.gz > ${WORKDIR}/GeoIPv6.dat");
> > + os.system("${FETCHCMD_wget} http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz");
> > + os.system("gunzip -c ${DL_DIR}/GeoLiteCity.dat.gz > ${WORKDIR}/GeoLiteCity.dat");
> > + os.system("${FETCHCMD_wget} http://geolite.maxmind.com/download/geoip/database/GeoLiteCityv6.dat.gz");
> > + os.system("gunzip -c ${DL_DIR}/GeoLiteCityv6.dat.gz > ${WORKDIR}/GeoLiteCityv6.dat");
> > +}
> > +
> > do_install() {
> > make DESTDIR=${D} install
> > install -d ${D}/${datadir}/GeoIP
> > --
> > 1.9.1
> >
> > --
> > _______________________________________________
> > Openembedded-devel mailing list
> > Openembedded-devel@lists.openembedded.org
> > http://lists.openembedded.org/mailman/listinfo/openembedded-devel
>
> --
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-devel
--
Martin 'JaMa' Jansa jabber: Martin.Jansa@gmail.com
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 188 bytes --]
^ permalink raw reply [flat|nested] 4+ messages in thread* Re: [meta-networking][PATCH] GeoIP: change data fetching strategy
2015-01-08 3:41 ` Khem Raj
2015-01-08 11:14 ` Martin Jansa
@ 2015-01-09 17:02 ` Joe MacDonald
1 sibling, 0 replies; 4+ messages in thread
From: Joe MacDonald @ 2015-01-09 17:02 UTC (permalink / raw)
To: openembedded-devel
[-- Attachment #1: Type: text/plain, Size: 5640 bytes --]
[Re: [oe] [meta-networking][PATCH] GeoIP: change data fetching strategy] On 15.01.07 (Wed 19:41) Khem Raj wrote:
>
> > On Jan 7, 2015, at 6:19 PM, Joe MacDonald <joe_macdonald@mentor.com> wrote:
> >
> > Since the GeoIP databases are not versioned and they essentially change
> > without notice, we cannot reasonably expect the checksums to be consistent
>
> isn’t this like AUTOREV ?.
Actually, yeah, it is. I didn't think this applied in this case since
we don't have a source repository for this, we can only get tarballs,
and therefore SRCREV wouldn't apply. I may dig into that a bit more
this afternoon and see if it's applicable.
> IMO such packages should be mirrored somewhere regularly and used. if
> you maintain this recipe then please mirror the sources somewhere and
> then update them regularly, but from one build to another is just
> too frequent.
That's true, it shouldn't be every build. As for mirroring them
somewhere, I suppose I can fetch updates and place them in a github
project, but if this is something we could place (and I can easily
update) on sources.openembedded.org, that may make more sense.
-J.
>
> > from one build to the next. Therefore, don't download them using the
> > usual wget fetcher, but do it manually at the end of the do_fetch stage
> > and place the uncompressed data files where the remainder of the build
> > expects they would've been placed.
> >
> > Signed-off-by: Joe MacDonald <joe_macdonald@mentor.com>
> > ---
> >
> > I considered trying to use the wget fetcher and having it skip over the
> > checksum operation and, honestly, I'd still prefer to do it that way, but
> > looking around and at the fetch2 code, it doesn't appear to be able to support
> > that outside of a BB_STRICT_CHECKSUM="" setting, which we definitely don't want
> > to do here.
> >
> > -J.
> >
> > .../recipes-support/geoip/geoip_1.6.0.bb | 36 +++++++++-------------
> > 1 file changed, 14 insertions(+), 22 deletions(-)
> >
> > diff --git a/meta-networking/recipes-support/geoip/geoip_1.6.0.bb b/meta-networking/recipes-support/geoip/geoip_1.6.0.bb
> > index 65e8362..71fda3b 100644
> > --- a/meta-networking/recipes-support/geoip/geoip_1.6.0.bb
> > +++ b/meta-networking/recipes-support/geoip/geoip_1.6.0.bb
> > @@ -8,28 +8,9 @@ using reverse DNS lookups."
> > HOMEPAGE = "http://dev.maxmind.com/geoip/"
> > SECTION = "Development/Libraries"
> >
> > -SRC_URI = "http://www.maxmind.com/download/geoip/api/c/GeoIP-1.6.0.tar.gz;name=tarball \
> > - http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz;apply=no;name=GeoIP-dat \
> > - http://geolite.maxmind.com/download/geoip/database/GeoIPv6.dat.gz;apply=no;name=GeoIPv6-dat \
> > - http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz;apply=no;name=GeoLiteCity-dat \
> > - http://geolite.maxmind.com/download/geoip/database/GeoLiteCityv6-beta/GeoLiteCityv6.dat.gz;apply=no;name=GeoLiteCityv6-dat \
> > -"
> > -
> > -SRC_URI[tarball.md5sum] = "89f4cdfdab43f1d67364cd7c85bbe8ca"
> > -SRC_URI[tarball.sha256sum] = "075a0c2815cd099e9ec35c9569db716a3fefcdbb6a10dbfa1ce7c6cd48d4a635"
> > -
> > -SRC_URI[GeoIP-dat.md5sum] = "4bc1e8280fe2db0adc3fe48663b8926e"
> > -SRC_URI[GeoIP-dat.sha256sum] = "7fd7e4829aaaae2677a7975eeecd170134195e5b7e6fc7d30bf3caf34db41bcd"
> > -
> > -SRC_URI[GeoIPv6-dat.md5sum] = "aac7e6e9b141de80934ecee52daf7f56"
> > -SRC_URI[GeoIPv6-dat.sha256sum] = "126fd2953eb193e60538e30b4465610530383f7a782745cacdca5ba6825f471c"
> > -
> > -SRC_URI[GeoLiteCity-dat.md5sum] = "15a42c684c53d2309e6632a6d6e02531"
> > -SRC_URI[GeoLiteCity-dat.sha256sum] = "5ec02a6d39d545c77ec12cc30c6a8856883d8f55522fc5cd4f25af80163c6b3c"
> > -
> > -SRC_URI[GeoLiteCityv6-dat.md5sum] = "49d6ec946fa0a2575b5112a68d71f933"
> > -SRC_URI[GeoLiteCityv6-dat.sha256sum] = "7a345e6cf0e59f8ab589ff15020241f0b03342dd04cc584f814c4f4700d49405"
> > -
> > +SRC_URI = "http://www.maxmind.com/download/geoip/api/c/GeoIP-1.6.0.tar.gz"
> > +SRC_URI[md5sum] = "89f4cdfdab43f1d67364cd7c85bbe8ca"
> > +SRC_URI[sha256sum] = "075a0c2815cd099e9ec35c9569db716a3fefcdbb6a10dbfa1ce7c6cd48d4a635"
> >
> > LICENSE = "LGPL-2.1"
> >
> > @@ -43,6 +24,17 @@ inherit autotools
> > EXTRA_OECONF = "--disable-static \
> > --disable-dependency-tracking "
> >
> > +do_fetch_append() {
> > + os.system("${FETCHCMD_wget} http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz");
> > + os.system("gunzip -c ${DL_DIR}/GeoIP.dat.gz > ${WORKDIR}/GeoIP.dat");
> > + os.system("${FETCHCMD_wget} http://geolite.maxmind.com/download/geoip/database/GeoIPv6.dat.gz");
> > + os.system("gunzip -c ${DL_DIR}/GeoIPv6.dat.gz > ${WORKDIR}/GeoIPv6.dat");
> > + os.system("${FETCHCMD_wget} http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz");
> > + os.system("gunzip -c ${DL_DIR}/GeoLiteCity.dat.gz > ${WORKDIR}/GeoLiteCity.dat");
> > + os.system("${FETCHCMD_wget} http://geolite.maxmind.com/download/geoip/database/GeoLiteCityv6.dat.gz");
> > + os.system("gunzip -c ${DL_DIR}/GeoLiteCityv6.dat.gz > ${WORKDIR}/GeoLiteCityv6.dat");
> > +}
> > +
> > do_install() {
> > make DESTDIR=${D} install
> > install -d ${D}/${datadir}/GeoIP
> > --
> > 1.9.1
> >
> > --
> > _______________________________________________
> > Openembedded-devel mailing list
> > Openembedded-devel@lists.openembedded.org
> > http://lists.openembedded.org/mailman/listinfo/openembedded-devel
>
--
-Joe MacDonald.
:wq
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 501 bytes --]
^ permalink raw reply [flat|nested] 4+ messages in thread