[PATCH 00/18] mirrors: enhance and add tons of mirrors

[Luis Chamberlain <mcgrof@kernel.org>]

If you are stuck in a world where you don't have much data for your
downloads you have only one option: mirror everything. So the motivation
for this is the exand the existing mirror support on kdevops to mirror
more things other than just the kernel and qemu. We now can mirror
*everything* we clone including kdevops itself.

We also enhance this all so that we can *use* the mirrors by *default*
if you have them and you are using local virtualization. We differentiate
between vagrant and guestfs.

Towards the end this just simplifies the process by adding a yaml file
for the mirrors so that we can ingest them and process that with a few
python scripts. The last commits demonstrate how easy it is now to add
new mirrors.

There's only few mirrors left to add but we can do that next week.

Luis Chamberlain (18):
  linux-mirror: add kdevops mirroring support
  Makefile.min_deps: add netcat as a min binary dependency
  provision: generalize bridge-ip further
  mirrors: add guestfs mirror options without 9p
  mirrors: enable kdevops mirror
  mirrors: add fstests to mirrors
  fstests: use local primary fstests mirror if present
  mirrors: add kdevops git alternative for fstests
  mirrors: use kdevops fstests mirror when available
  mirrors: add blktests mirror
  mirrors: use blktests mirror when its available
  mirrors: increase scope of mirroring in output
  mirrors: move status check
  mirrors: move mirror editing into one file
  mirrors: add new mirrors.yaml and add xfsprogs
  mirrors: add xfsdump mirror and use it
  mirrors: add dbench git
  mirrors: add blktrace mirror and use it

 .gitignore                                    |   5 +
 Kconfig                                       |   1 +
 Makefile                                      |   2 -
 Makefile.linux-mirror                         |  55 ++-----
 Makefile.min_deps                             |   1 +
 kconfigs/Kconfig.ansible_provisioning         |   4 +-
 kconfigs/Kconfig.defaults                     |  18 +++
 kconfigs/Kconfig.guestfs                      |   6 -
 kconfigs/Kconfig.mirror                       | 115 +++++++++++++-
 kconfigs/workflows/Kconfig.shared             |  11 +-
 playbooks/linux-mirror.yml                    |   4 +
 playbooks/roles/blktests/defaults/main.yml    |   1 +
 playbooks/roles/blktests/tasks/main.yml       |   2 +-
 playbooks/roles/fstests/defaults/main.yml     |   2 +
 .../tasks/install-deps/debian/main.yml        |   2 +-
 playbooks/roles/linux-mirror/README.md        |   9 ++
 .../roles/linux-mirror/defaults/main.yml      |   8 +
 .../linux-mirror-systemd/Makefile             |  62 +++++++-
 .../blktests-mirror.service                   |  13 ++
 .../blktests-mirror.timer                     |  11 ++
 .../fstests-mirror.service                    |  13 ++
 .../linux-mirror-systemd/fstests-mirror.timer |  11 ++
 .../kdevops-fstests-mirror.service            |  13 ++
 .../kdevops-fstests-mirror.timer              |  11 ++
 .../kdevops-mirror.service                    |  13 ++
 .../linux-mirror-systemd/kdevops-mirror.timer |  10 ++
 .../linux-mirror/python/gen-mirror-files.py   | 149 ++++++++++++++++++
 .../linux-mirror/python/start-mirroring.py    | 110 +++++++++++++
 playbooks/roles/linux-mirror/tasks/main.yml   |  91 +++++++++++
 .../linux-mirror/templates/mirrors.yaml.j2    |  13 ++
 scripts/append-makefile-vars.sh               |  10 ++
 scripts/check_mirror_present.sh               |  11 ++
 scripts/get-distro-default-bridge.sh          |  15 ++
 scripts/get-distro-prefix.sh                  |  18 +++
 scripts/provision.Makefile                    |  12 ++
 workflows/blktests/Kconfig                    |  29 +++-
 workflows/blktests/Makefile                   |   1 +
 workflows/fstests/Kconfig                     |  20 ++-
 workflows/fstests/Makefile.xfs                |   2 +
 workflows/fstests/xfs/Kconfig                 |  20 ++-
 workflows/linux/Kconfig.linus                 |   3 +-
 workflows/linux/Kconfig.mcgrof                |   2 +
 workflows/linux/Kconfig.next                  |   3 +-
 workflows/linux/Kconfig.stable                |   3 +-
 44 files changed, 852 insertions(+), 63 deletions(-)
 create mode 100644 kconfigs/Kconfig.defaults
 create mode 100644 playbooks/linux-mirror.yml
 create mode 100644 playbooks/roles/linux-mirror/README.md
 create mode 100644 playbooks/roles/linux-mirror/defaults/main.yml
 create mode 100644 playbooks/roles/linux-mirror/linux-mirror-systemd/blktests-mirror.service
 create mode 100644 playbooks/roles/linux-mirror/linux-mirror-systemd/blktests-mirror.timer
 create mode 100644 playbooks/roles/linux-mirror/linux-mirror-systemd/fstests-mirror.service
 create mode 100644 playbooks/roles/linux-mirror/linux-mirror-systemd/fstests-mirror.timer
 create mode 100644 playbooks/roles/linux-mirror/linux-mirror-systemd/kdevops-fstests-mirror.service
 create mode 100644 playbooks/roles/linux-mirror/linux-mirror-systemd/kdevops-fstests-mirror.timer
 create mode 100644 playbooks/roles/linux-mirror/linux-mirror-systemd/kdevops-mirror.service
 create mode 100644 playbooks/roles/linux-mirror/linux-mirror-systemd/kdevops-mirror.timer
 create mode 100755 playbooks/roles/linux-mirror/python/gen-mirror-files.py
 create mode 100755 playbooks/roles/linux-mirror/python/start-mirroring.py
 create mode 100644 playbooks/roles/linux-mirror/tasks/main.yml
 create mode 100644 playbooks/roles/linux-mirror/templates/mirrors.yaml.j2
 create mode 100755 scripts/append-makefile-vars.sh
 create mode 100755 scripts/check_mirror_present.sh
 create mode 100755 scripts/get-distro-default-bridge.sh
 create mode 100755 scripts/get-distro-prefix.sh

-- 
2.43.0