Re: Newbie questions

[Jesper Dangaard Brouer <brouer@redhat.com>]

On Mon, 21 Jun 2021 22:28:09 -0300
"Ethy H. Brito" <ethy.brito@inexo.com.br> wrote:

> On Fri, 18 Jun 2021 17:37:17 -0300
> "Ethy H. Brito" <ethy.brito@inexo.com.br> wrote:
> 
> > On Fri, 18 Jun 2021 19:40:17 +0200
> > Jesper Dangaard Brouer <brouer@redhat.com> wrote:
> >   
> > > On Fri, 18 Jun 2021 13:31:06 -0300
> > > "Ethy H. Brito" <ethy.brito@inexo.com.br> wrote:
> > >     
> > > > Hi All.
> > > > 
> > > > I've been doing some home work reading the docs and some doubts have raised.
> > > > For reference, my environment is 
> > > > 	Ubuntu 20.04
> > > > 	kernel 5.4.0-66 
> > > > 	tc utility, iproute2-ss200127.
> > > > 
> > > > 1) https://xdp-project.net/areas/cpumap.html#cpumap--Create-script-MQ-HTB-silo-setup says that: 
> > > > 	"XPS (Transmit Packet Steering) will take precedence over any changes to 
> > > > 	skb->queue_mapping. You need to disable *XDP* via mask=00 in files 
> > > > 	/sys/class/net/DEV/queues/tx-*/xps_cpus"
> > > > 
> > > > Shouldn't it say I need to disable *XPS* (not XDP) using mask=00??      
> > > 
> > > You are absolutely right it is a typo. Can I ask you to fix that and
> > > send a GitHub PR?
> > > 
> > > The file you need to change is:
> > >  https://github.com/xdp-project/xdp-project/blob/master/areas/cpumap.org  
> 
> File edited. PR sent.

Thanks merged it.

> > >     
> > > > How to set that CPU-0 will deal with mq queue 7FFF:1, CPU-1 will deal
> > > > with 7FFF:2, and so on?      
> > > 
> > > That is the role of the XDP program that redirect into a cpumap, and
> > > the key in the cpumap is the CPU number.    
> 
> OK. I see that in source code.

Yes, see the explanation in the source code.
Also read: https://github.com/xdp-project/xdp-cpumap-tc/blob/master/src/howto_debug.org

The "tc_queue_mapping_kern.c" program[1] is the simplest solution,
which only does the skb->queue_mapping, and you have to configure Linux
to set the correct TC minor:major number on a per packet basis (e.g.
via iptables see comment in code).

The "tc_classify_kern.c" program[2] is more advanced and have
implemented a IP-lookup map that have this[3] config per entry:

  struct ip_hash_info {
	/* lookup key: __u32 IPv4-address */
	__u32 cpu;
	__u32 tc_handle; /* TC handle MAJOR:MINOR combined in __u32 */
  };

 [1] https://github.com/xdp-project/xdp-cpumap-tc/blob/master/src/tc_queue_mapping_kern.c#L40-L76
 [2] https://github.com/xdp-project/xdp-cpumap-tc/blob/master/src/tc_classify_kern.c#L277
 [3] https://github.com/xdp-project/xdp-cpumap-tc/blob/master/src/common_kern_user.h#L29-L33

> But I am still pretty in the dark here to start using XDP.

Okay, then let me explain some basic concepts for xdp-cpumap-tc.

 1. XDP need to run on physical NIC with driver that supports (native) XDP.

 2. XDP is a layer before network stack, before the "SKB" is created.

 3. XDP redirect the raw frame to another CPU via XDP_REDIRECT'ing into a cpumap.

 4. The cpumap (kthread) running on remote CPU will create the SKB and
    call normal network stack on this CPU.

 5. The TC-BPF program running on remote CPU update skb->queue_mapping
    (and possibly skb->priority) to map packet into the TC-queue of
    your choosing.

Notice for you scenario there are 4 BPF-progs running, two XDP and two
TC-BPF.  See what is running via cmdline: "bpftool net"

 # bpftool net
 xdp:
 eno49(4) driver id 22
 eno50(5) driver id 26

 tc:
 eno49(4) clsact/egress tc_classify_kern.o:[tc_classify] id 42
 eno50(5) clsact/egress tc_classify_kern.o:[tc_classify] id 43

All the BPF-programs share BPF-maps to have the same config.

Maps pinned:
 # ls -1 /sys/fs/bpf/tc/globals/
 map_ifindex_type
 map_ip_hash
 map_txq_config


> More newbie questions are necessary.
> 
> My goal is simple: to control the bandwidth of a few (or a lot)
> thousands users using an of-the-shelf (almost) box. Two 10Gbps ether
> interface. One internal, one external.

I have access to a production system, that have 2x 25Gbit/s NIC (plus
VLANs for each apartment building), let me check how many customers
they have added.  They are using[2] "tc_classify_kern.c" and their
IP-map contains 6086 entries (more than I expected actually).

> What come in thru eth0 goes out to eth0 or eth1 and what comes in
> thru eth1 comes out to eth0. 
> 
> Is there a road map about what to execute and in what order to
> achieve this task using xdp-cpumap-tc?

This is already available today, and running in production at an ISP.
Sorry for the lack of documentation on how to use it, but it is done.

> I have cloned xdp-cpumap-tc to try figuring it out reading the source code. 
> But things did not get together.
> 
> For instance, tc_classify_kern.c (as tc_queue_mapping_kern.c)  "talks" about a "manuel" (sic)
> setup:
> 
> 	tc qdisc  add dev ixgbe2 clsact
> 	tc filter add dev ixgbe2 egress bpf da obj tc_classify_kern.o sec tc_classify
> 
> At what point these commands are to be executed?
> They are not mentioned anywhere else. (tc_mq_htb_setup_example.sh forgot these perhaps?)

This is handled by: tc_classify_user
 https://github.com/xdp-project/xdp-cpumap-tc/blob/master/src/tc_classify_user.c

The TC commands are called from C-code in this file:
 https://github.com/xdp-project/xdp-cpumap-tc/blob/master/src/common_user.c

The roadmap is to convert this to use the new libbpf TC API instead, as
it is a mess to have a dependency on the right iproute2 version.

 
> Which one is be to loaded tc_classify_kern or tc_queue_mapping_kern?
> Or both? None? After and before what?

Actually due to limitation in iproute2 loader, you should load
XDP-programs first (as it will create maps with BTF info).

You cannot load tc_classify_kern and tc_queue_mapping_kern simultaneously.

> 
> In the file tc_classify_kern.c, map_ifindex_type is defined
> differently from xdp_iphash_to_cpu_kern.c.
>
> 	".size_value = sizeof(struct txq_config)" in the former
> and  
> 	".size_value = sizeof(__u32)" int the later.
>
> Is this a "Cut and paste" typo? Are they really meant to be two
> different maps?

Hmm... this looks like a copy-paste error.  The tc_classify_kern.c
map_ifindex_type should have size_value = sizeof(__u32). It happens to
work because sizeof(struct txq_config) is also 4 bytes.


> Anyway, a step by step guide would be appreciated.

I'm hoping you will create/document that once you learn howto use these
programs ;-)

> Maybe it is time to start populating that BNG-router repo I was told about.
> How can I start helping with that? Worth doing it?

I think we need to convince other ISP's to join in...
... let me CC those guys again.

-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer