[Qemu-devel] Moving release tarballs to a CDN

[Qemu-devel] Moving release tarballs to a CDN

[Stefan Hajnoczi]

Hi Mike and Jeff,
qemu.org's bandwidth usage is dominated by release tarball downloads.
This puts qemu.org bandwidth usage in the 2+ TB/month range.

Many hosting providers do not offer this much bandwidth as part of a
basic package that would otherwise fit qemu.org's requirements.

Several people have suggested putting release tarballs on a mirror or
CDN to offload the bandwidth usage.  Jeff Cody and I looked at using
GitHub's "releases" feature to host release tarballs.

The following changes would be necessary:

1. Add qemu.org URL redirection to GitHub:
https://download.qemu.org/qemu-(\d+.\d+.\d+).tar.xz ->
https://github.com/qemu/qemu/releases/download/v\1/qemu-\1.tar.xz

2. Modify release script to immediately push to GitHub and then upload
tar.xz via API:
https://developer.github.com/v3/repos/releases/#upload-a-release-asset

3. Continue using https://download.qemu.org/qemu-\d+.\d+.\d+.tar.xz
URL for download links.

This way future QEMU releases would be hosted on GitHub with nice
qemu.org download links.  The signature file stays hosted on qemu.org
so it's possible to validate that the tarball has not been tampered
with.

How does this sound?

Stefan

Re: [Qemu-devel] Moving release tarballs to a CDN

[Stefan Weil]

Am 08.11.2017 um 16:33 schrieb Stefan Hajnoczi:
> Hi Mike and Jeff,
> qemu.org's bandwidth usage is dominated by release tarball downloads.
> This puts qemu.org bandwidth usage in the 2+ TB/month range.

Hi Stefan,

how much of this traffic is caused by web spiders?

>From my own binaries I know that the bots of the
different search engines cause most of the traffic,
if they are allowed to do so.

Usually they respect robots.txt. There is no
https://www.qemu.org/robots.txt currently.
Nor is there a https://download.qemu.org/robots.txt.
Adding both would reduce the downloads, maybe
enough to fix the problem.

Or do you see an advantage from bots which download
QEMU tarballs? robots.txt can also block only
selected bots.

Regards
Stefan

PS. There is a https://git.qemu.org/robots.txt.

Re: [Qemu-devel] Moving release tarballs to a CDN

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1088 bytes --]

On Wed, Nov 08, 2017 at 05:19:25PM +0100, Stefan Weil wrote:
> Am 08.11.2017 um 16:33 schrieb Stefan Hajnoczi:
> > Hi Mike and Jeff,
> > qemu.org's bandwidth usage is dominated by release tarball downloads.
> > This puts qemu.org bandwidth usage in the 2+ TB/month range.
> 
> Hi Stefan,
> 
> how much of this traffic is caused by web spiders?
> 
> From my own binaries I know that the bots of the
> different search engines cause most of the traffic,
> if they are allowed to do so.
> 
> Usually they respect robots.txt. There is no
> https://www.qemu.org/robots.txt currently.
> Nor is there a https://download.qemu.org/robots.txt.
> Adding both would reduce the downloads, maybe
> enough to fix the problem.
> 
> Or do you see an advantage from bots which download
> QEMU tarballs? robots.txt can also block only
> selected bots.
> 
> Regards
> Stefan
> 
> PS. There is a https://git.qemu.org/robots.txt.

Great idea!  It's an easy to try adding a robots.txt and check how
bandwidth uses changes over the next month.

Jeff: Want to try this?

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] Moving release tarballs to a CDN

[Jeff Cody]

On Wed, Nov 08, 2017 at 05:11:20PM +0000, Stefan Hajnoczi wrote:
> On Wed, Nov 08, 2017 at 05:19:25PM +0100, Stefan Weil wrote:
> > Am 08.11.2017 um 16:33 schrieb Stefan Hajnoczi:
> > > Hi Mike and Jeff,
> > > qemu.org's bandwidth usage is dominated by release tarball downloads.
> > > This puts qemu.org bandwidth usage in the 2+ TB/month range.
> > 
> > Hi Stefan,
> > 
> > how much of this traffic is caused by web spiders?
> > 
> > From my own binaries I know that the bots of the
> > different search engines cause most of the traffic,
> > if they are allowed to do so.
> > 
> > Usually they respect robots.txt. There is no
> > https://www.qemu.org/robots.txt currently.
> > Nor is there a https://download.qemu.org/robots.txt.
> > Adding both would reduce the downloads, maybe
> > enough to fix the problem.
> > 
> > Or do you see an advantage from bots which download
> > QEMU tarballs? robots.txt can also block only
> > selected bots.
> > 
> > Regards
> > Stefan
> > 
> > PS. There is a https://git.qemu.org/robots.txt.
> 
> Great idea!  It's an easy to try adding a robots.txt and check how
> bandwidth uses changes over the next month.
> 
> Jeff: Want to try this?
> 
> Stefan

Yes, sure - I added a robots.txt to exclude .bz2 and .xz files, and we can
see how that affects bandwidth.  Right now, with our current hosting provider,
we are not near any bandwidth limit, but it makes sense to conserve
resources (unless there is any benefit we see to allowing bots to index
download.qemu.org binaries).


Thanks,

Jeff