Re: [Qemu-devel] Moving release tarballs to a CDN

[Stefan Weil <stefan.weil@weilnetz.de>]

Am 08.11.2017 um 16:33 schrieb Stefan Hajnoczi:
> Hi Mike and Jeff,
> qemu.org's bandwidth usage is dominated by release tarball downloads.
> This puts qemu.org bandwidth usage in the 2+ TB/month range.

Hi Stefan,

how much of this traffic is caused by web spiders?

>From my own binaries I know that the bots of the
different search engines cause most of the traffic,
if they are allowed to do so.

Usually they respect robots.txt. There is no
https://www.qemu.org/robots.txt currently.
Nor is there a https://download.qemu.org/robots.txt.
Adding both would reduce the downloads, maybe
enough to fix the problem.

Or do you see an advantage from bots which download
QEMU tarballs? robots.txt can also block only
selected bots.

Regards
Stefan

PS. There is a https://git.qemu.org/robots.txt.