Re: ftp.kernel.org

[Chris Shoemaker <c.shoemaker@cox.net>]

On Fri, May 28, 2004 at 11:01:19AM -0400, Theodore Ts'o wrote:
> On Fri, May 28, 2004 at 02:21:41AM -0400, Chris Shoemaker wrote:
> > > Agreed - fmirror is so much more reliable than rsync (imho) that it makes 
> > > rsync into a worst-case option for retrieving files.
> > 
> >   bug reports to rsync@lists.samba.org are appreciated...
> > 
> 
> The main problem with rsync that I can see is that it is fairly heavy
> weight on the server, so many servers (including mirrors.kernel.org)
> have a maximum number of connections set to something pathetically
> small, like say 5 connections.  

Do you mean w.r.t. memory usage or storage i/o?  I know that creating
the file list can take up a lot of memory for large lists.  

Five connections seems pretty low.  I've never personally hit any
connection limit, and I make moderate use of rsync.  On the server side 
I know of several rsync servers offering >1 million files.  Not sure how 
hard they work, but they're highly available.

> 
> I remember Tridge trying to get someone to implement checksum caching
> for rsync servers some 4+ years ago, which would surely help.  Did
> that ever get done?  If so, convincing the server admins that it's OK
> to up the maximum number of rsync connections would be the next step.
> 
> 						- Ted

I'm sure there are some things that can be done to make rsync 
lighter-weight.  Checksums aren't cached, but the problem is, the 
checksum seed is varible, so that might not be the best optimization.

Overall, I'd have to disagree with the parent-post saying that rsync is
worst-case option.  It's not perfect, but I much prefer rsync over
fmirror.  I think it's more convenient and, although I have no rigorous
measurements, but it seems faster, to boot.

-chris