[PATCH xtables-addons 0/4] geoip: script fixes

[PATCH xtables-addons 0/4] geoip: script fixes

[Jeremy Sowden]

A couple of fixes and some man-pages for the MaxMind geoip scripts.

Jeremy Sowden (4):
  geoip: remove superfluous xt_geoip_fetch_maxmind script.
  geoip: fix man-page typo'.
  geoip: add man-pages for MaxMind scripts.
  geoip: use correct download URL for MaxMind DB's.

 geoip/Makefile.am              |  6 ++-
 geoip/xt_geoip_build_maxmind.1 | 40 ++++++++++++++
 geoip/xt_geoip_dl_maxmind      | 12 ++++-
 geoip/xt_geoip_dl_maxmind.1    | 22 ++++++++
 geoip/xt_geoip_fetch.1         |  2 +-
 geoip/xt_geoip_fetch_maxmind   | 95 ----------------------------------
 6 files changed, 77 insertions(+), 100 deletions(-)
 create mode 100644 geoip/xt_geoip_build_maxmind.1
 create mode 100644 geoip/xt_geoip_dl_maxmind.1
 delete mode 100755 geoip/xt_geoip_fetch_maxmind

-- 
2.29.2

[PATCH xtables-addons 1/4] geoip: remove superfluous xt_geoip_fetch_maxmind script.

[Jeremy Sowden]

xt_geoip_fetch and xt_geoip_fetch_maxmind are identical.  Remove the
latter.

Signed-off-by: Jeremy Sowden <jeremy@azazel.net>
---
 geoip/Makefile.am            |  2 +-
 geoip/xt_geoip_fetch_maxmind | 95 ------------------------------------
 2 files changed, 1 insertion(+), 96 deletions(-)
 delete mode 100755 geoip/xt_geoip_fetch_maxmind

diff --git a/geoip/Makefile.am b/geoip/Makefile.am
index 7bbf3bcf3815..5323c82eb7c4 100644
--- a/geoip/Makefile.am
+++ b/geoip/Makefile.am
@@ -1,6 +1,6 @@
 # -*- Makefile -*-
 
-bin_SCRIPTS = xt_geoip_fetch xt_geoip_fetch_maxmind
+bin_SCRIPTS = xt_geoip_fetch
 
 pkglibexec_SCRIPTS = xt_geoip_build xt_geoip_build_maxmind xt_geoip_dl xt_geoip_dl_maxmind
 
diff --git a/geoip/xt_geoip_fetch_maxmind b/geoip/xt_geoip_fetch_maxmind
deleted file mode 100755
index 06245195fb51..000000000000
--- a/geoip/xt_geoip_fetch_maxmind
+++ /dev/null
@@ -1,95 +0,0 @@
-#!/usr/bin/perl
-#
-#	Utility to query GeoIP database
-#	Copyright Philip Prindeville, 2018
-#
-use Getopt::Long;
-use Socket qw(AF_INET AF_INET6 inet_ntop);
-use warnings;
-use strict;
-
-sub AF_INET_SIZE() { 4 }
-sub AF_INET6_SIZE() { 16 }
-
-my $target_dir = ".";
-my $ipv4 = 0;
-my $ipv6 = 0;
-
-&Getopt::Long::Configure(qw(bundling));
-&GetOptions(
-	"D=s" => \$target_dir,
-	"4"   => \$ipv4,
-	"6"   => \$ipv6,
-);
-
-if (!-d $target_dir) {
-	print STDERR "Target directory $target_dir does not exit.\n";
-	exit 1;
-}
-
-# if neither specified, assume both
-if (! $ipv4 && ! $ipv6) {
-	$ipv4 = $ipv6 = 1;
-}
-
-foreach my $cc (@ARGV) {
-	if ($cc !~ m/^([a-z]{2}|a[12]|o1)$/i) {
-		print STDERR "Invalid country code '$cc'\n";
-		exit 1;
-	}
-
-	my $file = $target_dir . '/' . uc($cc) . '.iv4';
-
-	if (! -f $file) {
-		printf STDERR "Can't find data for country '$cc'\n";
-		exit 1;
-	}
-
-	my ($contents, $buffer, $bytes, $fh);
-
-	if ($ipv4) {
-		open($fh, '<', $file) || die "Couldn't open file for '$cc'\n";
-
-		binmode($fh);
-
-		while (($bytes = read($fh, $buffer, AF_INET_SIZE * 2)) == AF_INET_SIZE * 2) {
-			my ($start, $end) = unpack('a4a4', $buffer);
-			$start = inet_ntop(AF_INET, $start);
-			$end = inet_ntop(AF_INET, $end);
-			print $start, '-', $end, "\n";
-		}
-		close($fh);
-		if (! defined $bytes) {
-			printf STDERR "Error reading file for '$cc'\n";
-			exit 1;
-		} elsif ($bytes != 0) {
-			printf STDERR "Short read on file for '$cc'\n";
-			exit 1;
-		}
-	}
-
-	substr($file, -1) = '6';
-
-	if ($ipv6) {
-		open($fh, '<', $file) || die "Couldn't open file for '$cc'\n";
-
-		binmode($fh);
-
-		while (($bytes = read($fh, $buffer, AF_INET6_SIZE * 2)) == AF_INET6_SIZE * 2) {
-			my ($start, $end) = unpack('a16a16', $buffer);
-			$start = inet_ntop(AF_INET6, $start);
-			$end = inet_ntop(AF_INET6, $end);
-			print $start, '-', $end, "\n";
-		}
-		close($fh);
-		if (! defined $bytes) {
-			printf STDERR "Error reading file for '$cc'\n";
-			exit 1;
-		} elsif ($bytes != 0) {
-			printf STDERR "Short read on file for '$cc'\n";
-			exit 1;
-		}
-	}
-}
-
-exit 0;
-- 
2.29.2

[PATCH xtables-addons 2/4] geoip: fix man-page typo'.

[Jeremy Sowden]

Signed-off-by: Jeremy Sowden <jeremy@azazel.net>
---
 geoip/xt_geoip_fetch.1 | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/geoip/xt_geoip_fetch.1 b/geoip/xt_geoip_fetch.1
index 7280c74b9ab5..5d1ae48ae42e 100644
--- a/geoip/xt_geoip_fetch.1
+++ b/geoip/xt_geoip_fetch.1
@@ -9,7 +9,7 @@ xt_geoip_fetch \(em dump a country database to stdout
 .SH Description
 .PP
 xt_geoip_fetch unpacks a country's IPv4 or IPv6 databases and dumps
-them to standard output as a sorted, non-overlaping list of ranges (which
+them to standard output as a sorted, non-overlapping list of ranges (which
 is how they're represented in the database) suitable for browsing or
 further processing.
 .PP Options
-- 
2.29.2

[PATCH xtables-addons 3/4] geoip: add man-pages for MaxMind scripts.

[Jeremy Sowden]

Signed-off-by: Jeremy Sowden <jeremy@azazel.net>
---
 geoip/Makefile.am              |  4 +++-
 geoip/xt_geoip_build_maxmind.1 | 40 ++++++++++++++++++++++++++++++++++
 geoip/xt_geoip_dl_maxmind.1    | 22 +++++++++++++++++++
 3 files changed, 65 insertions(+), 1 deletion(-)
 create mode 100644 geoip/xt_geoip_build_maxmind.1
 create mode 100644 geoip/xt_geoip_dl_maxmind.1

diff --git a/geoip/Makefile.am b/geoip/Makefile.am
index 5323c82eb7c4..8c0b6af80054 100644
--- a/geoip/Makefile.am
+++ b/geoip/Makefile.am
@@ -4,4 +4,6 @@ bin_SCRIPTS = xt_geoip_fetch
 
 pkglibexec_SCRIPTS = xt_geoip_build xt_geoip_build_maxmind xt_geoip_dl xt_geoip_dl_maxmind
 
-man1_MANS = xt_geoip_build.1 xt_geoip_dl.1 xt_geoip_fetch.1
+man1_MANS = xt_geoip_build.1 xt_geoip_dl.1 \
+	    xt_geoip_build_maxmind.1 xt_geoip_dl_maxmind.1 \
+	    xt_geoip_fetch.1
diff --git a/geoip/xt_geoip_build_maxmind.1 b/geoip/xt_geoip_build_maxmind.1
new file mode 100644
index 000000000000..e20e44848b82
--- /dev/null
+++ b/geoip/xt_geoip_build_maxmind.1
@@ -0,0 +1,40 @@
+.TH xt_geoip_build_maxmind 1 "2010-12-17" "xtables-addons" "xtables-addons"
+.SH Name
+.PP
+xt_geoip_build_maxmind \(em convert GeoIP.csv to packed format for xt_geoip
+.SH Syntax
+.PP
+\fI/usr/libexec/xt_geoip/\fP\fBxt_geoip_build_maxmind\fP [\fB\-D\fP
+\fItarget_dir\fP] [\fB\-S\fP \fIsource_dir\fP]
+.SH Description
+.PP
+xt_geoip_build_maxmind is used to build packed raw representations of the range
+database that the xt_geoip module relies on. Since kernel memory is precious,
+much of the preprocessing is done in userspace by this very building tool. One
+file is produced for each country, so that no more addresses than needed are
+required to be loaded into memory. The ranges in the packed database files are
+also ordered, as xt_geoip relies on this property for its bisection approach to
+work.
+.PP
+Since the script is usually installed to the libexec directory of the
+xtables-addons package and this is outside $PATH (on purpose), invoking the
+script requires it to be called with a path.
+.PP Options
+.TP
+\fB\-D\fP \fItarget_dir\fP
+Specifies the target directory into which the files are to be put. Defaults to ".".
+.TP
+\fB\-S\fP \fIsource_dir\fP
+Specifies the source directory of the MaxMind CSV files. Defaults to ".".
+.TP
+\fB\-s\fP
+"System mode". Equivalent to \fB\-D /usr/share/xt_geoip\fP.
+.SH Application
+.PP
+Shell commands to build the databases and put them to where they are expected
+(usually run as root):
+.PP
+xt_geoip_build_maxmind \-s
+.SH See also
+.PP
+xt_geoip_dl_maxmind(1)
diff --git a/geoip/xt_geoip_dl_maxmind.1 b/geoip/xt_geoip_dl_maxmind.1
new file mode 100644
index 000000000000..00a73d7ee90d
--- /dev/null
+++ b/geoip/xt_geoip_dl_maxmind.1
@@ -0,0 +1,22 @@
+.TH xt_geoip_dl_maxmind 1 "2010-12-17" "xtables-addons" "xtables-addons"
+.SH Name
+.PP
+xt_geoip_dl_maxmind \(em download MaxMind GeoIP database files
+.SH Syntax
+.PP
+\fI/usr/libexec/xt_geoip/\fP\fBxt_geoip_dl_maxmind\fP [\fI licence-key file\fP]
+.SH Description
+.PP
+Downloads the MaxMind GeoLite2 databases for IPv4 and IPv6 and unpacks them to
+the current directory.  The alternate \fBxt_geoip_dl\fP script can be
+used for the DB-IP Country Lite databases.
+.PP
+Since the script is usually installed to the libexec directory of the
+xtables-addons package and this is outside $PATH (on purpose), invoking the
+script requires it to be called with a path.
+.SH Options
+.PP
+None.
+.SH See also
+.PP
+xt_geoip_build_maxmind(1)
-- 
2.29.2

[PATCH xtables-addons 4/4] geoip: use correct download URL for MaxMind DB's.

[Jeremy Sowden]

The download URL for the GeoLite2 DB's has changed and includes a
licence-key.  Update the download script to read the key from file or
stdin and use the correct URL.

Signed-off-by: Jeremy Sowden <jeremy@azazel.net>
---
 geoip/xt_geoip_dl_maxmind | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/geoip/xt_geoip_dl_maxmind b/geoip/xt_geoip_dl_maxmind
index 1de60442a804..d5640336c1c0 100755
--- a/geoip/xt_geoip_dl_maxmind
+++ b/geoip/xt_geoip_dl_maxmind
@@ -1,7 +1,16 @@
 #!/bin/sh
 
+if [ $# -eq 1 ]; then
+    exec <$1
+elif [ $# -ne 0 ]; then
+    echo $(basename $0) [ licence_key_file ] 1>&2
+    exit 1
+fi
+
+read licence_key
+
 rm -rf GeoLite2-Country-CSV_*
 
-wget -q http://geolite.maxmind.com/download/geoip/database/GeoLite2-Country-CSV.zip
+wget -q -OGeoLite2-Country-CSV.zip "https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-Country-CSV&license_key=${licence_key}&suffix=zip"
 unzip -q GeoLite2-Country-CSV.zip
 rm -f GeoLite2-Country-CSV.zip
-- 
2.29.2

Re: [PATCH xtables-addons 2/4] geoip: fix man-page typo'.

[Jan Engelhardt]

On Sunday 2020-11-22 15:05, Jeremy Sowden wrote:
>Subject: geoip: fix man-page typo'.
>Signed-off-by: Jeremy Sowden <jeremy@azazel.net>

Rhetoric: How much more interpunction do you want to add to the subject 
lines?
Neither apostrophe nor period belong into it.

Re: [PATCH xtables-addons 0/4] geoip: script fixes

[Jan Engelhardt]

On Sunday 2020-11-22 15:05, Jeremy Sowden wrote:

>A couple of fixes and some man-pages for the MaxMind geoip scripts.
>
>Jeremy Sowden (4):
>  geoip: remove superfluous xt_geoip_fetch_maxmind script.
>  geoip: fix man-page typo'.
>  geoip: add man-pages for MaxMind scripts.
>  geoip: use correct download URL for MaxMind DB's.

Applied.

Re: [PATCH xtables-addons 0/4] geoip: script fixes

[Jeremy Sowden]

[-- Attachment #1: Type: text/plain, Size: 548 bytes --]

On 2020-11-22, at 17:55:03 +0100, Jan Engelhardt wrote:
> On Sunday 2020-11-22 15:05, Jeremy Sowden wrote:
> > A couple of fixes and some man-pages for the MaxMind geoip scripts.
> >
> > Jeremy Sowden (4):
> >   geoip: remove superfluous xt_geoip_fetch_maxmind script.
> >   geoip: fix man-page typo'.
> >   geoip: add man-pages for MaxMind scripts.
> >   geoip: use correct download URL for MaxMind DB's.
>
> Applied.

Thanks!  I only see 1-3 in your tree, however.  Was there something
wrong with the fourth patch or did it just get mislaid?

J.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH xtables-addons 0/4] geoip: script fixes

[Jan Engelhardt]

On Tuesday 2020-11-24 18:29, Jeremy Sowden wrote:
>On 2020-11-22, at 17:55:03 +0100, Jan Engelhardt wrote:
>> On Sunday 2020-11-22 15:05, Jeremy Sowden wrote:
>> > Jeremy Sowden (4):
>> >   geoip: remove superfluous xt_geoip_fetch_maxmind script.
>> >   geoip: fix man-page typo'.
>> >   geoip: add man-pages for MaxMind scripts.
>> >   geoip: use correct download URL for MaxMind DB's.
>>
>> Applied.
>
>Thanks!  I only see 1-3 in your tree, however.

Indeed, now it's there.