Re: [libvirt] [PATCH 2/2] libxl: support vif outgoing bandwidth QoS

[Michal Privoznik <mprivozn@redhat.com>]

On 08.01.2016 06:11, Jim Fehlig wrote:
> On 01/07/2016 07:48 AM, Michal Privoznik wrote:
>> On 29.12.2015 02:09, Jim Fehlig wrote:
>>> The libxl_device_nic structure supports specifying an outgoing rate
>>> limit based on a time interval and bytes allowed per interval. In xl
>>> config a rate limit is specified as "<RATE>/s@<INTERVAL>". INTERVAL
>>> is optional and defaults to 50ms.
>>>
>>> libvirt expresses outgoing limits by average (required), peak, burst,
>>> and floor attributes in units of KB/s. This patch supports the outgoing
>>> bandwidth limit by converting the average KB/s to bytes per interval
>>> based on the same default interval (50ms) used by xl.
>>>
>>> Signed-off-by: Jim Fehlig <jfehlig@suse.com>
>>> ---
>>>  src/libxl/libxl_conf.c | 39 +++++++++++++++++++++++++++++++++++++++
>>>  1 file changed, 39 insertions(+)
>>>
>>> diff --git a/src/libxl/libxl_conf.c b/src/libxl/libxl_conf.c
>>> index 23c74e7..6320421 100644
>>> --- a/src/libxl/libxl_conf.c
>>> +++ b/src/libxl/libxl_conf.c
>>> @@ -1093,6 +1093,7 @@ libxlMakeNic(virDomainDefPtr def,
>>>  {
>>>      bool ioemu_nic = def->os.type == VIR_DOMAIN_OSTYPE_HVM;
>>>      virDomainNetType actual_type = virDomainNetGetActualType(l_nic);
>>> +    virNetDevBandwidthPtr actual_bw;
>>>  
>>>      /* TODO: Where is mtu stored?
>>>       *
>>> @@ -1206,6 +1207,44 @@ libxlMakeNic(virDomainDefPtr def,
>>>  #endif
>>>      }
>>>  
>>> +    /*
>>> +     * Set bandwidth.
>>> +     * From $xen-sources/docs/misc/xl-network-configuration.markdown:
>>> +     *
>>> +     *
>>> +     * Specifies the rate at which the outgoing traffic will be limited to.
>>> +     * The default if this keyword is not specified is unlimited.
>>> +     *
>>> +     * The rate may be specified as "<RATE>/s" or optionally "<RATE>/s@<INTERVAL>".
>>> +     *
>>> +     * `RATE` is in bytes and can accept suffixes:
>>> +     *     GB, MB, KB, B for bytes.
>>> +     *     Gb, Mb, Kb, b for bits.
>>> +     * `INTERVAL` is in microseconds and can accept suffixes: ms, us, s.
>>> +     *     It determines the frequency at which the vif transmission credit
>>> +     *     is replenished. The default is 50ms.
>>> +
>>> +     * Vif rate limiting is credit-based. It means that for "1MB/s@20ms",
>>> +     * the available credit will be equivalent of the traffic you would have
>>> +     * done at "1MB/s" during 20ms. This will results in a credit of 20,000
>>> +     * bytes replenished every 20,000 us.
>>> +     *
>>> +     *
>>> +     * libvirt doesn't support the notion of rate limiting over an interval.
>>> +     * Similar to xl's behavior when interval is not specified, set a default
>>> +     * interval of 50ms and calculate the number of bytes per interval based
>>> +     * on the specified average bandwidth.
>>> +     */
>>> +    actual_bw = virDomainNetGetActualBandwidth(l_nic);
>>> +    if (actual_bw && actual_bw->out && actual_bw->out->average) {
>>> +        uint64_t bytes_per_sec = actual_bw->out->average * 1024;
>>> +        uint64_t bytes_per_interval =
>>> +            (((uint64_t) bytes_per_sec * 50000UL) / 1000000UL);
>>> +
>>> +        x_nic->rate_bytes_per_interval = bytes_per_interval;
>>> +        x_nic->rate_interval_usecs =  50000UL;
>>> +    }
>>> +
>> Interesting. I'd expect:
>>
>> x_nic->rate_bytes_per_interval = bytes_per_sec;
>> x_nic->rate_interval_usecs = 1000*1000;
> 
> For the most part I mimicked the Xen code and wanted to stick with the default
> interval of 50ms, which has been the default for a long time. It is even
> mentioned in some old RHEL5 docs
> 
> https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/5/html/Virtualization/sect-Virtualization-Tips_and_tricks-Limit_network_bandwidth_for_a_Xen_guest.html
> 
> BTW, here is the Xen code that inspired this logic
> 
> http://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=tools/libxl/libxlu_vif.c;h=0665e624dc178a6ca8058e04a7baacaf1475bd37;hb=HEAD#l131
> 
> rate_bytes_per_interval is set to (bytes/s * interval us)/1000000us
> 
> I guess we are saying the same thing, you're just setting interval to 1s (thus
> rate_bytes_per_interval == bytes_per_sec) instead of the historical 50ms :-).
> 
>>
>> I mean, if I understood the xl way of rate limiting correctly, one says
>> how much bytes can be sent for how long. so for 1MB/s I'd expect to send
>> 1024*1024 bytes each second.
>>
>> Or am I missing something?
> 
> Does the above explanation make sense? I might be missing something :-).  CC'd a
> few Xen tools maintainer just in case.


Yep, it makes sense now. So let me review v2.

Michal