Re: Limit bandwidth per-user (uid/gid)

[Paride Legovini <pl@ninthfloor.org>]

Dear Carl-Daniel,

thanks for your reply, it contains a lot of wise suggestions.

On Mon, Dec 23, 2013 at 04:22:08AM +0100, Carl-Daniel Hailfinger wrote:
> Dear Paride,
> 
> that's a really exciting environment.
> 
> Am 22.12.2013 18:10 schrieb Paride Legovini:
> > I'm working in an Antarctic research station where our connection to the
> > Internet is a 512kbps satellite link.
> 
> Assuming that the satellite link is expensive and has long roundtrip
> times, it is probably a wise idea not to throw away any data which has
> already been sent over the satellite link.

You're perfectly right here: the link is expensive and the round trip is
700ms at best, but reaches easily 3000ms or even 4000ms when the network
is congested. 

> > I want to set up a server where each research project has an account
> > where they send data via sftp or rsync; this data is then transferred
> > overnight to a server in Europe. My idea is to use a separate cronjob
> > or daemon for each user that runs with the user's privileges.
> >
> > What I want to do is:
> >
> > 1. Limit the total bandwidth that a group (GID) can generate. There
> >    should be separate limits for inbound and outbound traffic.
> 
> Shaping outbound traffic is mostly a standard procedure with lots of
> documentation available. Nowadays you have some more options than a few
> years ago, and quite a few of them are easy to set up. I'll conveniently
> ignore outbound shaping in my response and talk about the hard stuff
> instead.
> You can't limit inbound traffic to Antactica unless you do it on the
> other end of the satellite link (Europe). Well, you can limit the
> inbound bandwidth by throwing away packets locally, but that's stupid
> for packets which have already come over an expensive satellite link.
> Only the remote side decides how much bandwidth to send over the
> satellite link. What has been sent by the remote side will go over the
> satellite link and you can't undo this on the local side.
> What you can do in Antarctica is:
> 1. Perform the transfer with bandwidth-aware tools (rsync has the
> bwlimit option, but that is too crude to be useful).
> 2. Rely on crude implicit+indirect throttling of inbound traffic by
> shaping the bandwidth passed to local applications, which will cause a
> sort of backpressure (changed Ack behaviour) once the kernel buffers for
> your connection are full. Some people also recommend to throttle/police
> outbound Acks, but with current TCP congestion control mechanisms this
> can have unintended side effects.
> 
> 
> > 2. Limit the bandwidth per-user (UID), so if the GID is allowed to
> >    generate 384kbps of traffic, and 3 users are using the network, each
> >    user can at most benefit of 128kbps. If there's only one user he gets
> >    all the 384kbps.
> >    Again there should be different limits for inbound and outbound
> >    traffic.
> >    This should work regardless the number of connections the user makes.
> >
> > I played a bit with iptables and tc, but the only way I found to do
> > something like this is to manually set a different mark for each user
> > and then use tc, but I'd prefer a solution where there's no need to set
> > up any rule manually if a user is added or removed. Also, the
> > --uid-owner option works only for outbound traffic.
> 
> AFAICS even if you use net_cls as suggested by Joseph Santaniello,
> you're limited to managing outbound traffic (net_cls may indeed a
> cleaner approach than uid-owner, I haven't tried that). With CONNMARK,
> you should be able to tie the pieces together. I did such a setup
> (CONNMARK, not net_cls) for inbound+outbound throttling quite a few
> years ago, but sadly the code has been lost somewhere in the depths of
> decommissioned harddisks.

If I understand correctly, this corresponds to your second suggestion
regarding the inbound traffic throttling. The difficult thing here is
that I not only have to mark a connection, but I have to mark all the
connection started by a certain user (cgroup) in the same (known) way,
so I can use tc on them.

I see that for inbound traffic shaping Intermediate Functional Block
(ifb) devices can also be used, but I don't think it will help me to do
a per-user (cgroup) throttling or shaping for the incoming traffic.
This seems a difficult task.

I'll think about it, but at first I think I'll concentrate on the
outgoing traffic.

Sunny greetings,

Paride