Other computers HIGHLY degrading network performance (DoS?)

Other computers HIGHLY degrading network performance (DoS?)

[Anuradha Ratnaweera]

This is not a direct kernel issse.  However, it is a serious threat for the
network performance of our Linux boxes, therefore I thought of posting it here.

There is a popular software that runs on MS platform called "download
accelerator".  This opens several threads for a download job (each one
downloading a portion of the file), sometimes even using mirror sites.
However, it not only grabs whole bandwidth, but makes it hard for other
machines to even ping each other the return time being around 5-10 seconds on a
100 Mbps network!  The download process is getting only 64 kbps from the
Internet.  Internet access is virtually impossible for the other machines.

This program can run with a `normal download' mode and this doesn't cause a
big problem.

I monitored network traffic with tcpdump, and noticed that those packets don't
have tcp timestamps and tcp sack.  I turned them off on my Linux box using
sysctl, and also tried turning on ECN without success.

This is of course a DoS in disguise, and is there a way to stop it?

I am thinking of setting up a firewall with netfilter and transparent proxy as
a workaround.

Thanks in advance.

Regards,

Anuradha

-- 

Debian GNU/Linux (kernel 2.4.13)

History books which contain no lies are extremely dull.

Re: Other computers HIGHLY degrading network performance (DoS?)

[Jeffrey H. Ingber]

I think this is what QoS and the like are for.

Jeffrey H. Ingber (jhingber _at_ ix.netcom.com)

On Thu, 2001-10-25 at 22:43, Anuradha Ratnaweera wrote:
> 
> This is not a direct kernel issse.  However, it is a serious threat for the
> network performance of our Linux boxes, therefore I thought of posting it here.
> 
> There is a popular software that runs on MS platform called "download
> accelerator".  This opens several threads for a download job (each one
> downloading a portion of the file), sometimes even using mirror sites.
> However, it not only grabs whole bandwidth, but makes it hard for other
> machines to even ping each other the return time being around 5-10 seconds on a
> 100 Mbps network!  The download process is getting only 64 kbps from the
> Internet.  Internet access is virtually impossible for the other machines.
> 
> This program can run with a `normal download' mode and this doesn't cause a
> big problem.
> 
> I monitored network traffic with tcpdump, and noticed that those packets don't
> have tcp timestamps and tcp sack.  I turned them off on my Linux box using
> sysctl, and also tried turning on ECN without success.
> 
> This is of course a DoS in disguise, and is there a way to stop it?
> 
> I am thinking of setting up a firewall with netfilter and transparent proxy as
> a workaround.
> 
> Thanks in advance.
> 
> Regards,
> 
> Anuradha
> 
> -- 
> 
> Debian GNU/Linux (kernel 2.4.13)
> 
> History books which contain no lies are extremely dull.
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: Other computers HIGHLY degrading network performance (DoS?)

[Anuradha Ratnaweera]

On Thu, Oct 25, 2001 at 10:55:16PM -0400, Jeffrey H. Ingber wrote:
>
> I think this is what QoS and the like are for.

Well, we _are_ going to solve the problem using a firewall between the router
and the local area network.

But the real problem is a different one.

One machine begins an intensive downloading job.  How can this degrade the
network performance even for ICMP packets between another machine and the
router?  Notice that this can't be collitions because the download goes at
64kbps and the local network is 100 Mbps.  Something funny is going on to
stop other people's packets.

Cheers,

Anuradha


-- 

Debian GNU/Linux (kernel 2.4.13)

True leadership is the art of changing a group from what it is to what
it ought to be.
		-- Virginia Allan

Re: Other computers HIGHLY degrading network performance (DoS?)

[Anuradha Ratnaweera]

On Fri, Oct 26, 2001 at 09:05:05AM +0600, Anuradha Ratnaweera wrote:
> On Thu, Oct 25, 2001 at 10:55:16PM -0400, Jeffrey H. Ingber wrote:
> >
> > I think this is what QoS and the like are for.
> 
> Well, we _are_ going to solve the problem using a firewall between the router
> and the local area network.
> 
> But the real problem is a different one.
> 
> One machine begins an intensive downloading job.  How can this degrade the
> network performance even for ICMP packets between another machine and the
> router?  Notice that this can't be collitions because the download goes at
> 64kbps and the local network is 100 Mbps.  Something funny is going on to
> stop other people's packets.

Just found out that this is _not_ a problem of the "download accelerator", but
something to do with queuing algorithm of the router.  Even a normal wget
process or a big mail has a big impart on the network.  Hopefully an iptables
firewall would solve the problem.

Thanks.

Regards,

Anuradha

-- 

Debian GNU/Linux (kernel 2.4.13)

Television has brought back murder into the home -- where it belongs.
	-- Alfred Hitchcock

Re: Other computers HIGHLY degrading network performance (DoS?)

[Paul P Komkoff Jr]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: RIPEMD160

Replying to Anuradha Ratnaweera:
> Just found out that this is _not_ a problem of the "download accelerator", but
> something to do with queuing algorithm of the router.  Even a normal wget
> process or a big mail has a big impart on the network.  Hopefully an iptables
> firewall would solve the problem.

Router hardware // software ??

- -- 
Paul P 'Stingray' Komkoff 'Greatest' Jr // (icq)23200764 // (irc)Spacebar
  PPKJ1-RIPE // (smtp)i@stingr.net // (http)stingr.net // (pgp)0xA4B4ECA4
-----BEGIN PGP SIGNATURE-----

iEYEAREDAAYFAjvZQVQACgkQyMW8naS07KTxqwCcD5lpDgNhuqBEP4b+fseQm2SW
qgwAoL3te/ab255BmfUCrQqmD+Uair56
=+ydi
-----END PGP SIGNATURE-----

Re: Other computers HIGHLY degrading network performance (DoS?)

[Anuradha Ratnaweera]

On Fri, Oct 26, 2001 at 02:56:21PM +0400, Paul P Komkoff Jr wrote:
> 
> Replying to Anuradha Ratnaweera:
> > Just found out that this is _not_ a problem of the "download accelerator", but
> > something to do with queuing algorithm of the router.  Even a normal wget
> > process or a big mail has a big impart on the network.  Hopefully an iptables
> > firewall would solve the problem.
> 
> Router hardware // software ??

Software.

It is a linux based propiatory product, and we don't have control over it.

Regards,

Anuradha

-- 

Debian GNU/Linux (kernel 2.4.13)

The easiest way to figure the cost of living is to take your income and
add ten percent.

Re: Other computers HIGHLY degrading network performance (DoS?)

[Paul P Komkoff Jr]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: RIPEMD160

Replying to Anuradha Ratnaweera:
> It is a linux based propiatory product, and we don't have control over it.

Then pay attention to other's posts and check thru your network.
Maybe some of devices connected to hub have full duplex on and kill the
whole (your) network (collision domain)

maybe it is router itself, maybe some of your users think they're smart and
'why don't use fd if my nic offer this feature'.

I've fought these idiots some time ago. But my network include switches and
if any idiot connected to hub wish to play with duplex - then he only play
with his own collision domain - not larger then single department.

and if it is not duplex issue, then take lan tester and test your cabling
... then hub ...
then get rid of your "proprietary linux based product"
:)

- -- 
Paul P 'Stingray' Komkoff 'Greatest' Jr // (icq)23200764 // (irc)Spacebar
  PPKJ1-RIPE // (smtp)i@stingr.net // (http)stingr.net // (pgp)0xA4B4ECA4
-----BEGIN PGP SIGNATURE-----

iEYEAREDAAYFAjvahzkACgkQyMW8naS07KQc1wCaA+V7mh80bIA8JEOCOtdvt5xn
PNYAoKQmotBJHdG1ri14yijbl6b7pgs8
=v/1K
-----END PGP SIGNATURE-----

Re: Other computers HIGHLY degrading network performance (DoS?)

[Anuradha Ratnaweera]

On Sat, Oct 27, 2001 at 02:06:51PM +0400, Paul P Komkoff Jr wrote:
> 
> and if it is not duplex issue, then take lan tester and test your cabling ...
> then hub ...

It _is_ probably a duplex issue.  We have already tested cabling and hub.

> then get rid of your "proprietary linux based product" :)
                  ^^^^
It is _not_ ours.  The router is controlled by the ISP.  Otherwise, we would
have got rid of it long time ago ;)

Cheers,

Anuradha

-- 

Debian GNU/Linux (kernel 2.4.13)

	"What time is it?"
	"I don't know, it keeps changing."

Re: Other computers HIGHLY degrading network performance (DoS?)

[Martin Josefsson]

On Fri, 26 Oct 2001, Anuradha Ratnaweera wrote:

> > One machine begins an intensive downloading job.  How can this degrade the
> > network performance even for ICMP packets between another machine and the
> > router?  Notice that this can't be collitions because the download goes at
> > 64kbps and the local network is 100 Mbps.  Something funny is going on to
> > stop other people's packets.
> 
> Just found out that this is _not_ a problem of the "download accelerator", but
> something to do with queuing algorithm of the router.  Even a normal wget
> process or a big mail has a big impart on the network.  Hopefully an iptables
> firewall would solve the problem.

I'd advice you to seriously look over your network, are you 100% sure you
don't have a duplex-issue anywhere?

I've been running linuxrouters for quite a while and right now I have a few
linuxrouters routing 100Mbit/s internetconnections. We have never had any
problems like the one you describe so my first guess would be that you
have a duplexproblem, probably between the linuxrouter and the switch it's
connected to on the inside, that's usually where it's located. Or maybe
between some switches or something but do look into this. I seriously
doubt that this a problem with the networking in linux.

/Martin

Never argue with an idiot. They drag you down to their level, then beat you with experience.

Re: Other computers HIGHLY degrading network performance (DoS?)

[Anuradha Ratnaweera]

On Fri, Oct 26, 2001 at 04:01:29PM +0200, Martin Josefsson wrote:
> On Fri, 26 Oct 2001, Anuradha Ratnaweera wrote:
> 
> > > One machine begins an intensive downloading job.  How can this degrade the
> > > network performance even for ICMP packets between another machine and the
> > > router?  Notice that this can't be collitions because the download goes at
> > > 64kbps and the local network is 100 Mbps.  Something funny is going on to
> > > stop other people's packets.
> > 
> > Just found out that this is _not_ a problem of the "download accelerator", but
> > something to do with queuing algorithm of the router.  Even a normal wget
> > process or a big mail has a big impart on the network.  Hopefully an iptables
> > firewall would solve the problem.
> 
> I'd advice you to seriously look over your network, are you 100% sure you
> don't have a duplex-issue anywhere?

I will double check.  I wonder if this is the cause, because the network is 100
Mbps, but the router is switching only at 64kbps.

> I've been running linuxrouters for quite a while and right now I have a few
> linuxrouters routing 100Mbit/s internetconnections. We have never had any
> problems like the one you describe so my first guess would be that you have a
> duplexproblem, probably between the linuxrouter and the switch it's connected
> to on the inside, that's usually where it's located.

We have a hub, and not a switch.  Can this be the reason?  BTW, how come that a
duplex issue can result in such huge degradations?

> I seriously doubt that this a problem with the networking in linux.

Not in linux, may be the way they have _used_ linux on the router ;-)

Anuradha

-- 

Debian GNU/Linux (kernel 2.4.13)

[FORTRAN] will persist for some time -- probably for at least the next decade.
		-- T. Cheatham

Re: Other computers HIGHLY degrading network performance (DoS?)

[Martin Josefsson]

On Sat, 27 Oct 2001, Anuradha Ratnaweera wrote:

> > > Just found out that this is _not_ a problem of the "download accelerator", but
> > > something to do with queuing algorithm of the router.  Even a normal wget
> > > process or a big mail has a big impart on the network.  Hopefully an iptables
> > > firewall would solve the problem.
> > 
> > I'd advice you to seriously look over your network, are you 100% sure you
> > don't have a duplex-issue anywhere?
> 
> I will double check.  I wonder if this is the cause, because the network is 100
> Mbps, but the router is switching only at 64kbps.

Our network almost died when someone changed the duplex in the switch
here...

> > I've been running linuxrouters for quite a while and right now I have a few
> > linuxrouters routing 100Mbit/s internetconnections. We have never had any
> > problems like the one you describe so my first guess would be that you have a
> > duplexproblem, probably between the linuxrouter and the switch it's connected
> > to on the inside, that's usually where it's located.
> 
> We have a hub, and not a switch.  Can this be the reason?  BTW, how come that a
> duplex issue can result in such huge degradations?

If you are trying to run full-duplex against the hub it till completely
kill the performance of the entire hub, been there, done that. When I did
that one I could only push 5-50kB/s through the hub.

> > I seriously doubt that this a problem with the networking in linux.
> 
> Not in linux, may be the way they have _used_ linux on the router ;-)

Hehe, I think you'll have to have quite weird QoS rules to manage to do
something like this :)

/Martin

Never argue with an idiot. They drag you down to their level, then beat you with experience.

Re: Other computers HIGHLY degrading network performance (DoS?)

[Alan Cox]

> There is a popular software that runs on MS platform called "download
> accelerator".  This opens several threads for a download job (each one
> downloading a portion of the file), sometimes even using mirror sites.
> However, it not only grabs whole bandwidth, but makes it hard for other
> machines to even ping each other the return time being around 5-10 seconds on a
> 100 Mbps network!  The download process is getting only 64 kbps from the
> Internet.  Internet access is virtually impossible for the other machines.

Firewall them off. There are also apache hacks for spotting the 
download accelerator device and blocking the user for good.

> I monitored network traffic with tcpdump, and noticed that those packets don't
> have tcp timestamps and tcp sack.  I turned them off on my Linux box using
> sysctl, and also tried turning on ECN without success.

They will tend to come from older windows boxes, the timestamp/sack stuff
is unrelated

> This is of course a DoS in disguise, and is there a way to stop it?

Turning off byte range support in the web server works suprisingly well for
it. Another non hacking code approach would be to set up CBQ or other
bandwidth limiters so that the users of download accelerator get no
benefit. The advantage of the apache hacks is that you can make them
actually suffer

Re: Other computers HIGHLY degrading network performance (DoS?)

[Anuradha Ratnaweera]

On Fri, Oct 26, 2001 at 01:53:08PM +0100, Alan Cox wrote:
> 
> Turning off byte range support in the web server works suprisingly well for
> it.

But won't it affect retry features of other `good' software such as wget?  At
any rate, people are downloading from web/ftp sites on the Internet, which are
beyond our control, and apache hacks are anyway impossible.  However, I am
thinking of setting up a transparent proxy (squid/iptables) on the firewall and
let it handle the bandwidth limiting and the like.

> Another non hacking code approach would be to set up CBQ or other bandwidth
> limiters so that the users of download accelerator get no benefit.

I don't mind hacking approaches.  Otherwise, I won't be reading the LKML ;-)

> The advantage of the apache hacks is that you can make them actually suffer

That is a wonderful idea.  Trying to exploit features on the Internet should be
a punishable offence ;)

Regards,

Anuradha

-- 

Debian GNU/Linux (kernel 2.4.13)

Exhilaration is that feeling you get just after a great idea hits you,
and just before you realize what is wrong with it.