XDP Performance Regression in recent kernel versions

XDP Performance Regression in recent kernel versions

[Sebastiano Miano]

Hi folks,

I have been conducting some basic experiments with XDP and have
observed a significant performance regression in recent kernel
versions compared to v5.15.

My setup is the following:
- Hardware: Two machines connected back-to-back with 100G Mellanox
ConnectX-6 Dx.
- DUT: 2x16 core Intel(R) Xeon(R) Silver 4314 CPU @ 2.40GHz.
- Software: xdp-bench program from [1] running on the DUT in both DROP
and TX modes.
- Traffic generator: Pktgen-DPDK sending traffic with a single 64B UDP
flow at ~130Mpps.
- Tests: Single core, HT disabled

Results:

Kernel version |-------| XDP_DROP |--------|   XDP_TX  |
5.15                                30Mpps                  16.1Mpps
6.2                                21.3Mpps                 14.1Mpps
6.5                                19.9Mpps                  8.6Mpps
bpf-next (6.10-rc2)        22.1Mpps                 9.2Mpps

I repeated the experiments multiple times and consistently obtained
similar results.
Are you aware of any performance regressions in recent kernel versions
that could explain these results?

[1] https://github.com/xdp-project/xdp-tools

Re: XDP Performance Regression in recent kernel versions

[Tariq Toukan]

On 18/06/2024 18:28, Sebastiano Miano wrote:
> Hi folks,
> 
> I have been conducting some basic experiments with XDP and have
> observed a significant performance regression in recent kernel
> versions compared to v5.15.
> 

Hi,

> My setup is the following:
> - Hardware: Two machines connected back-to-back with 100G Mellanox
> ConnectX-6 Dx.
> - DUT: 2x16 core Intel(R) Xeon(R) Silver 4314 CPU @ 2.40GHz.
> - Software: xdp-bench program from [1] running on the DUT in both DROP
> and TX modes.
> - Traffic generator: Pktgen-DPDK sending traffic with a single 64B UDP
> flow at ~130Mpps.
> - Tests: Single core, HT disabled
> 
> Results:
> 
> Kernel version |-------| XDP_DROP |--------|   XDP_TX  |
> 5.15                                30Mpps                  16.1Mpps
> 6.2                                21.3Mpps                 14.1Mpps
> 6.5                                19.9Mpps                  8.6Mpps
> bpf-next (6.10-rc2)        22.1Mpps                 9.2Mpps
> 
> I repeated the experiments multiple times and consistently obtained
> similar results.
> Are you aware of any performance regressions in recent kernel versions
> that could explain these results?
> 
> [1] https://github.com/xdp-project/xdp-tools
> 

Thanks for your report.

I assume cpu util for the active core on the DUT is 100% in all cases, 
right?

Can you please share some more details? Like relevant ethtool counters, 
and perf top output.

We'll check if this repro for us as well.

Re: XDP Performance Regression in recent kernel versions

[Sebastiano Miano]

On Wed, 19 Jun 2024 at 08:00, Tariq Toukan <tariqt@nvidia.com> wrote:
>
> Thanks for your report.
>
> I assume cpu util for the active core on the DUT is 100% in all cases,
> right?

Yes, that's correct.
The irq is also on the core on the right numa node, and I have
disabled CPU frequency scaling.

>
> Can you please share some more details? Like relevant ethtool counters,
> and perf top output.
>
> We'll check if this repro for us as well.

Sure, below you can find the reports for the XDP_DROP and XDP_TX cases.
I am attaching only the ones for kern v5.15 vs v6.5.

--------------------------------------------------
ethtool output (5.15) - Missing counters are zero
--------------------------------------------------
NIC statistics:
     rx_packets: 333854100
     rx_bytes: 20031246044
     tx_packets: 25
     tx_bytes: 2070
     rx_csum_unnecessary: 333854079
     rx_xdp_drop: 3753342954
     rx_xdp_redirect: 0
     rx_xdp_tx_xmit: 5582660674
     rx_xdp_tx_mpwqe: 175018775
     rx_xdp_tx_inlnw: 8970048
     rx_xdp_tx_nops: 378338337
     rx_xdp_tx_full: 0
     rx_xdp_tx_err: 0
     rx_xdp_tx_cqe: 87229072
     rx_cache_reuse: 9369255040
     rx_cache_full: 68
     rx_cache_empty: 16153471
     rx_cache_busy: 193
     rx_cache_waive: 15864256
     rx_congst_umr: 158
     ch_events: 448
     ch_poll: 151091830
     ch_arm: 301
     rx_out_of_buffer: 990473555
     rx_if_down_packets: 67469721
     rx_steer_missed_packets: 1962570491
     rx_vport_unicast_packets: 38460159194
     rx_vport_unicast_bytes: 2461450188460
     tx_vport_unicast_packets: 5582654212
     tx_vport_unicast_bytes: 334959252764
     tx_packets_phy: 5588396729
     rx_packets_phy: 97052087562
     tx_bytes_phy: 357657403514
     rx_bytes_phy: 6211329423080
     tx_mac_control_phy: 5745055
     tx_pause_ctrl_phy: 5745055
     rx_discards_phy: 58591428329
     tx_discards_phy: 0
     tx_errors_phy: 0
     rx_undersize_pkts_phy: 0
     rx_fragments_phy: 0
     rx_jabbers_phy: 0
     rx_64_bytes_phy: 97052040472
     rx_65_to_127_bytes_phy: 3
     rx_128_to_255_bytes_phy: 0
     rx_256_to_511_bytes_phy: 26
     rx_512_to_1023_bytes_phy: 0
     rx_1024_to_1518_bytes_phy: 0
     rx_1519_to_2047_bytes_phy: 0
     rx_2048_to_4095_bytes_phy: 0
     rx_4096_to_8191_bytes_phy: 0
     rx_8192_to_10239_bytes_phy: 0
     rx_prio0_bytes: 6211318150440
     rx_prio0_packets: 38460533605
     rx_prio0_discards: 58591314012
     tx_prio0_bytes: 357288052986
     tx_prio0_packets: 5582625883
     tx_global_pause: 5745042
     tx_global_pause_duration: 771103810
     ch0_events: 55
     ch0_poll: 146981606
     ch0_arm: 35
     ch0_aff_change: 6
     ch0_force_irq: 0
     ch0_eq_rearm: 0
     rx0_packets: 70812690
     rx0_bytes: 4248761400
     rx0_csum_complete: 0
     rx0_csum_complete_tail: 0
     rx0_csum_complete_tail_slow: 0
     rx0_csum_unnecessary: 70812671
     rx0_csum_unnecessary_inner: 0
     rx0_csum_none: 19
     rx0_xdp_drop: 3753342954
     rx0_xdp_redirect: 0
     rx0_lro_packets: 0
     rx0_lro_bytes: 0
     rx0_ecn_mark: 0
     rx0_removed_vlan_packets: 0
     rx0_wqe_err: 0
     rx0_mpwqe_filler_cqes: 0
     rx0_mpwqe_filler_strides: 0
     rx0_oversize_pkts_sw_drop: 0
     rx0_buff_alloc_err: 0
     rx0_cqe_compress_blks: 0
     rx0_cqe_compress_pkts: 0
     rx0_cache_reuse: 9368316609
     rx0_cache_full: 2
     rx0_cache_empty: 11519
     rx0_cache_busy: 0
     rx0_cache_waive: 0
     rx0_congst_umr: 158
     rx0_arfs_err: 0
     rx0_recover: 0
     rx0_xdp_tx_xmit: 5582664928
     rx0_xdp_tx_mpwqe: 175018908
     rx0_xdp_tx_inlnw: 8970048
     rx0_xdp_tx_nops: 378338623
     rx0_xdp_tx_full: 0
     rx0_xdp_tx_err: 0
     rx0_xdp_tx_cqes: 87229139

--------------------------------------------------
perf top output (5.15) - XDP_DROP
--------------------------------------------------
19.27%  [kernel]                  [k] mlx5e_skb_from_cqe_mpwrq_linear
11.74%  [kernel]                  [k] mlx5e_handle_rx_cqe_mpwrq
9.82%   [kernel]                  [k] mlx5e_xdp_handle
9.43%   [kernel]                  [k] mlx5e_alloc_rx_mpwqe
9.29%   bpf_prog_xdp_basic_prog   [k] bpf_prog_5f76c01f0ff23233_xdp_basic_prog
7.06%   [kernel]                  [k] mlx5e_page_release_dynamic
6.95%   [kernel]                  [k] mlx5e_poll_rx_cq
5.89%   [kernel]                  [k] dma_sync_single_for_cpu
5.21%   [kernel]                  [k] dma_sync_single_for_device
4.12%   [kernel]                  [k] mlx5e_free_rx_mpwqe
1.65%   [kernel]                  [k] mlx5e_poll_ico_cq
1.60%   [kernel]                  [k] mlx5e_napi_poll
1.59%   [kernel]                  [k] bpf_get_smp_processor_id
0.94%   [kernel]                  [k] bpf_dispatcher_xdp_func
0.91%   [kernel]                  [k] net_rx_action
0.90%   bpf_prog_xdp_dispatcher   [k] bpf_prog_17d608957d1f805a_xdp_dispatcher
0.90%   [kernel]                  [k] bpf_dispatcher_xdp
0.64%   [kernel]                  [k] mlx5e_post_rx_mpwqes
0.64%   [kernel]                  [k] mlx5e_poll_xdpsq_cq
0.37%   [kernel]                  [k] __softirqentry_text_start

--------------------------------------------------
perf top output (5.15) - XDP_TX
--------------------------------------------------
13.84%  bpf_prog_xdp_swap_macs_prog  [k]
bpf_prog_0a3ad412f28cbb6d_xdp_swap_macs_prog
11.43%  [kernel]                     [k] mlx5e_xmit_xdp_buff
10.69%  [kernel]                     [k] mlx5e_skb_from_cqe_mpwrq_linear
9.79%  [kernel]                      [k] mlx5e_xmit_xdp_frame_mpwqe
8.35%  [kernel]                      [k] mlx5e_handle_rx_cqe_mpwrq
6.34%  [kernel]                      [k] dma_sync_single_for_device
6.20%  [kernel]                      [k] mlx5e_poll_rx_cq
5.62%  [kernel]                      [k] mlx5e_page_release_dynamic
5.33%  [kernel]                      [k] mlx5e_xdp_handle
5.21%  [kernel]                      [k] mlx5e_alloc_rx_mpwqe
4.47%  [kernel]                      [k] mlx5e_free_xdpsq_desc
3.26%  [kernel]                      [k] dma_sync_single_for_cpu
1.47%  [kernel]                      [k] mlx5e_xmit_xdp_frame_check_mpwqe
1.22%  [kernel]                      [k] mlx5e_poll_xdpsq_cq
0.95%  [kernel]                      [k] net_rx_action
0.90%  [kernel]                      [k] bpf_get_smp_processor_id
0.80%  [kernel]                      [k] mlx5e_napi_poll
0.69%  [kernel]                      [k] mlx5e_xdp_mpwqe_session_start
0.63%  [kernel]                      [k] mlx5e_poll_ico_cq
0.49%  [kernel]                      [k] bpf_dispatcher_xdp
0.47%  [kernel]                      [k] bpf_dispatcher_xdp_func

---------------------------------------------------------------------------------------

--------------------------------------------------
ethtool output (6.5) - Missing counters are zero
--------------------------------------------------
NIC statistics:
     rx_packets: 7282880
     rx_bytes: 436973482
     tx_packets: 42
     tx_bytes: 3556
     rx_csum_unnecessary: 7282816
     rx_xdp_drop: 7783331724
     rx_xdp_redirect: 0
     rx_xdp_tx_xmit: 46956452544
     rx_xdp_tx_mpwqe: 4401807536
     rx_xdp_tx_inlnw: 46951234092
     rx_xdp_tx_nops: 4988835176
     rx_xdp_tx_full: 0
     rx_xdp_tx_err: 0
     rx_xdp_tx_cqe: 733694572
     rx_pp_alloc_fast: 3641784
     rx_pp_alloc_slow: 8
     rx_pp_alloc_slow_high_order: 0
     rx_pp_alloc_empty: 8
     rx_pp_alloc_refill: 0
     rx_pp_alloc_waive: 0
     rx_pp_recycle_cached: 3641280
     ch_events: 505
     ch_poll: 855423286
     rx_out_of_buffer: 534918379
     rx_if_down_packets: 4044804
     rx_steer_missed_packets: 298
     rx_vport_unicast_packets: 287214261626
     rx_vport_unicast_bytes: 18381712744116
     tx_vport_unicast_packets: 46956452544
     tx_vport_unicast_bytes: 2817387157674
     tx_packets_phy: 47000866603
     rx_packets_phy: 728277471186
     tx_bytes_phy: 3008055468662
     rx_bytes_phy: 46609758231313
     tx_mac_control_phy: 44414017
     tx_pause_ctrl_phy: 44414017
     rx_discards_phy: 441063206498
     rx_64_bytes_phy: 728277470842
     rx_65_to_127_bytes_phy: 133
     rx_128_to_255_bytes_phy: 0
     rx_256_to_511_bytes_phy: 211
     rx_512_to_1023_bytes_phy: 0
     rx_1024_to_1518_bytes_phy: 0
     rx_1519_to_2047_bytes_phy: 0
     rx_2048_to_4095_bytes_phy: 0
     rx_4096_to_8191_bytes_phy: 0
     rx_8192_to_10239_bytes_phy: 0
     rx_buffer_passed_thres_phy: 1192226
     rx_prio0_bytes: 46609758231313
     rx_prio0_packets: 287214264688
     rx_prio0_discards: 441063206498
     tx_prio0_bytes: 3005212971574
     tx_prio0_packets: 46956452586
     tx_global_pause: 44414017
     tx_global_pause_duration: 5961284324
     ch0_events: 120
     ch0_poll: 855423025
     ch0_arm: 100
     ch0_aff_change: 0
     ch0_force_irq: 0
     ch0_eq_rearm: 0
     rx0_packets: 7282880
     rx0_bytes: 436973482
     rx0_csum_complete: 0
     rx0_csum_complete_tail: 0
     rx0_csum_complete_tail_slow: 0
     rx0_csum_unnecessary: 7282816
     rx0_csum_unnecessary_inner: 0
     rx0_csum_none: 64
     rx0_xdp_drop: 7783331724
     rx0_xdp_redirect: 0
     rx0_lro_packets: 0
     rx0_lro_bytes: 0
     rx0_gro_packets: 0
     rx0_gro_bytes: 0
     rx0_gro_skbs: 0
     rx0_gro_match_packets: 0
     rx0_gro_large_hds: 0
     rx0_ecn_mark: 0
     rx0_removed_vlan_packets: 0
     rx0_wqe_err: 0
     rx0_mpwqe_filler_cqes: 0
     rx0_mpwqe_filler_strides: 0
     rx0_oversize_pkts_sw_drop: 0
     rx0_buff_alloc_err: 0
     rx0_cqe_compress_blks: 0
     rx0_cqe_compress_pkts: 0
     rx0_congst_umr: 0
     rx0_arfs_err: 0
     rx0_recover: 0
     rx0_pp_alloc_fast: 3641784
     rx0_pp_alloc_slow: 8
     rx0_pp_alloc_slow_high_order: 0
     rx0_pp_alloc_empty: 8
     rx0_pp_alloc_refill: 0
     rx0_pp_alloc_waive: 0
     rx0_pp_recycle_cached: 3641280
     rx0_pp_recycle_cache_full: 0
     rx0_pp_recycle_ring: 0
     rx0_pp_recycle_ring_full: 0
     rx0_pp_recycle_released_ref: 0
     rx0_xdp_tx_xmit: 46956452544
     rx0_xdp_tx_mpwqe: 4401807536
     rx0_xdp_tx_inlnw: 46951234092
     rx0_xdp_tx_nops: 4988835176
     rx0_xdp_tx_full: 0
     rx0_xdp_tx_err: 0
     rx0_xdp_tx_cqes: 733694572

--------------------------------------------------
perf top output (6.5) - XDP_DROP
--------------------------------------------------
27.63%  [kernel]                [k] mlx5e_skb_from_cqe_mpwrq_linear
12.61%  [kernel]                [k] mlx5e_handle_rx_cqe_mpwrq
8.38%  [kernel]                 [k] mlx5e_rx_cq_process_basic_cqe_comp
7.06%  [kernel]                 [k] page_pool_put_defragged_page
6.45%  [kernel]                 [k] mlx5e_xdp_handle
5.36%  bpf_prog_xdp_basic_prog  [k] bpf_prog_5f76c01f0ff23233_xdp_basic_prog
4.95%  [kernel]                 [k] dma_sync_single_for_device
4.89%  [kernel]                 [k] page_pool_alloc_pages
4.36%  [kernel]                 [k] mlx5e_alloc_rx_mpwqe
3.70%  [kernel]                 [k] dma_sync_single_for_cpu
2.71%  [kernel]                 [k] mlx5e_page_release_fragmented.isra.0
2.09%  [kernel]                 [k] bpf_dispatcher_xdp_func
1.95%  [kernel]                 [k] mlx5e_free_rx_mpwqe
1.10%  [kernel]                 [k] mlx5e_poll_ico_cq
1.07%  [kernel]                 [k] bpf_get_smp_processor_id
1.05%  [kernel]                 [k] mlx5e_napi_poll
0.85%  [kernel]                 [k] mlx5e_poll_xdpsq_cq
0.61%  [kernel]                 [k] net_rx_action
0.58%  bpf_prog_xdp_dispatcher  [k] bpf_prog_17d608957d1f805a_xdp_dispatcher
0.57%  [kernel]                 [k] bpf_dispatcher_xdp
0.53%  [kernel]                 [k] mlx5e_post_rx_mpwqes
0.27%  [kernel]                 [k] __do_softirq
0.25%  [kernel]                 [k] mlx5e_poll_tx_cq

--------------------------------------------------
perf top output (6.5) - XDP_TX
--------------------------------------------------
19.60%  [kernel]                    [k] mlx5e_xdp_mpwqe_add_dseg
14.61%  [kernel]                    [k] mlx5e_skb_from_cqe_mpwrq_linear
11.55%  [kernel]                    [k] mlx5e_xmit_xdp_buff
5.85%  [kernel]                     [k] mlx5e_handle_rx_cqe_mpwrq
5.73%  bpf_prog_xdp_swap_macs_prog  [k] bpf_prog_0a3a_xdp_swap_macs_prog
5.09%  [kernel]                     [k] mlx5e_free_xdpsq_desc
5.08%  [kernel]                     [k] dma_sync_single_for_device
4.66%  [kernel]                     [k] mlx5e_xmit_xdp_frame_mpwqe
3.64%  [kernel]                     [k] mlx5e_rx_cq_process_basic_cqe_comp
3.34%  [kernel]                     [k] page_pool_put_defragged_page
3.04%  [kernel]                     [k] mlx5e_xdp_handle
3.03%  [kernel]                     [k] mlx5e_page_release_fragmented.isra.0
2.56%  [kernel]                     [k] dma_sync_single_for_cpu
2.15%  [kernel]                     [k] mlx5e_alloc_rx_mpwqe
1.96%  [kernel]                     [k] page_pool_alloc_pages
1.06%  [kernel]                     [k] mlx5e_xmit_xdp_frame_check_mpwqe
1.02%  [kernel]                     [k] bpf_dispatcher_xdp_func
1.01%  [kernel]                     [k] mlx5e_free_rx_mpwqe
0.84%  [kernel]                     [k] mlx5e_poll_xdpsq_cq
0.62%  [kernel]                     [k] mlx5e_xdpsq_get_next_pi
0.53%  [kernel]                     [k] mlx5e_poll_ico_cq
0.48%  [kernel]                     [k] bpf_get_smp_processor_id
0.48%  [kernel]                     [k] net_rx_action
0.36%  [kernel]                     [k] mlx5e_napi_poll
0.32%  [kernel]                     [k] mlx5e_xdp_mpwqe_complete
0.25%  [kernel]                     [k] bpf_dispatcher_xdp
0.22%  bpf_prog_xdp_dispatcher      [k] bpf_prog_17d6_xdp_dispatcher
0.21%  [kernel]                     [k] mlx5e_post_rx_mpwqes
0.11%  [kernel]                     [k] __do_softirq

Re: XDP Performance Regression in recent kernel versions

[Jesper Dangaard Brouer]

On 18/06/2024 17.28, Sebastiano Miano wrote:
> Hi folks,
> 
> I have been conducting some basic experiments with XDP and have
> observed a significant performance regression in recent kernel
> versions compared to v5.15.
> 
> My setup is the following:
> - Hardware: Two machines connected back-to-back with 100G Mellanox
> ConnectX-6 Dx.
> - DUT: 2x16 core Intel(R) Xeon(R) Silver 4314 CPU @ 2.40GHz.
> - Software: xdp-bench program from [1] running on the DUT in both DROP
> and TX modes.
> - Traffic generator: Pktgen-DPDK sending traffic with a single 64B UDP
> flow at ~130Mpps.
> - Tests: Single core, HT disabled
> 
> Results:
> 
> Kernel version |-------| XDP_DROP |--------|   XDP_TX  |
> 5.15                      30Mpps               16.1Mpps
> 6.2                       21.3Mpps             14.1Mpps
> 6.5                       19.9Mpps              8.6Mpps
> bpf-next (6.10-rc2)       22.1Mpps              9.2Mpps
> 

Around when I left Red Hat there were a project with [LNST] that used
xdp-bench for tracking and finding regressions like this.

Perhaps Toke can enlighten us, if that project have caught similar 
regressions?

[LNST] https://github.com/LNST-project/lnst


> I repeated the experiments multiple times and consistently obtained
> similar results.
> Are you aware of any performance regressions in recent kernel versions
> that could explain these results?
> 
> [1] https://github.com/xdp-project/xdp-tools


--Jesper

Re: XDP Performance Regression in recent kernel versions

[Toke Høiland-Jørgensen]

Jesper Dangaard Brouer <hawk@kernel.org> writes:

> On 18/06/2024 17.28, Sebastiano Miano wrote:
>> Hi folks,
>> 
>> I have been conducting some basic experiments with XDP and have
>> observed a significant performance regression in recent kernel
>> versions compared to v5.15.
>> 
>> My setup is the following:
>> - Hardware: Two machines connected back-to-back with 100G Mellanox
>> ConnectX-6 Dx.
>> - DUT: 2x16 core Intel(R) Xeon(R) Silver 4314 CPU @ 2.40GHz.
>> - Software: xdp-bench program from [1] running on the DUT in both DROP
>> and TX modes.
>> - Traffic generator: Pktgen-DPDK sending traffic with a single 64B UDP
>> flow at ~130Mpps.
>> - Tests: Single core, HT disabled
>> 
>> Results:
>> 
>> Kernel version |-------| XDP_DROP |--------|   XDP_TX  |
>> 5.15                      30Mpps               16.1Mpps
>> 6.2                       21.3Mpps             14.1Mpps
>> 6.5                       19.9Mpps              8.6Mpps
>> bpf-next (6.10-rc2)       22.1Mpps              9.2Mpps
>> 
>
> Around when I left Red Hat there were a project with [LNST] that used
> xdp-bench for tracking and finding regressions like this.
>
> Perhaps Toke can enlighten us, if that project have caught similar 
> regressions?
>
> [LNST] https://github.com/LNST-project/lnst

Yes, actually, we have! Here's the bugzilla for it:
https://bugzilla.redhat.com/show_bug.cgi?id=2270408

I'm on PTO for the rest of this week, but adding Samuel who ran the
tests to Cc, he should be able to provide more information if needed.

-Toke

Re: XDP Performance Regression in recent kernel versions

[Daniel Borkmann]

On 6/19/24 9:17 PM, Toke Høiland-Jørgensen wrote:
> Jesper Dangaard Brouer <hawk@kernel.org> writes:
>> On 18/06/2024 17.28, Sebastiano Miano wrote:
>>> I have been conducting some basic experiments with XDP and have
>>> observed a significant performance regression in recent kernel
>>> versions compared to v5.15.
>>>
>>> My setup is the following:
>>> - Hardware: Two machines connected back-to-back with 100G Mellanox
>>> ConnectX-6 Dx.
>>> - DUT: 2x16 core Intel(R) Xeon(R) Silver 4314 CPU @ 2.40GHz.
>>> - Software: xdp-bench program from [1] running on the DUT in both DROP
>>> and TX modes.
>>> - Traffic generator: Pktgen-DPDK sending traffic with a single 64B UDP
>>> flow at ~130Mpps.
>>> - Tests: Single core, HT disabled
>>>
>>> Results:
>>>
>>> Kernel version |-------| XDP_DROP |--------|   XDP_TX  |
>>> 5.15                      30Mpps               16.1Mpps
>>> 6.2                       21.3Mpps             14.1Mpps
>>> 6.5                       19.9Mpps              8.6Mpps
>>> bpf-next (6.10-rc2)       22.1Mpps              9.2Mpps
>>>
>>
>> Around when I left Red Hat there were a project with [LNST] that used
>> xdp-bench for tracking and finding regressions like this.
>>
>> Perhaps Toke can enlighten us, if that project have caught similar
>> regressions?
>>
>> [LNST] https://github.com/LNST-project/lnst
> 
> Yes, actually, we have! Here's the bugzilla for it:
> https://bugzilla.redhat.com/show_bug.cgi?id=2270408

 > We compared performance of ELN and RHEL9 candidate kernels and noticed significant
 > drop in XDP drop [1] on mlx5 (25G).
 >
 > On any rhel9 candidate kernel we are able to drop 19-20M pkts/sec but on an ELN
 > kernels, we are reaching just 15M pkts/sec (CPU utillization remains the same -
 > around 100%).
 >
 > We don't see such regression on ixgbe or i40e.

It looks like this is known since March, was this ever reported to Nvidia back
then? :/

Given XDP is in the critical path for many in production, we should think about
regular performance reporting for the different vendors for each released kernel,
similar to here [0].

Out of curiosity, @Saeed: Is Nvidia internally regularly assessing XDP perf for mlx5
as part of QA? (Probably not, but I thought I'd ask.)

Thanks,
Daniel

   [0] http://core.dpdk.org/perf-reports/

Re: XDP Performance Regression in recent kernel versions

[Samuel Dobron]

Hey all,

Yeah, we do tests for ELN kernels [1] on a regular basis. Since
~January of this year.

As already mentioned, mlx5 is the only driver affected by this regression.
Unfortunately, I think Jesper is actually hitting 2 regressions we noticed,
the one already mentioned by Toke, another one [0] has been reported
in early February.
Btw. issue mentioned by Toke has been moved to Jira, see [5].

Not sure all of you are able to see the content of [0], Jira says it's
RH-confidental.
So, I am not sure how much I can share without being fired :D. Anyway,
affected kernels have been released a while ago, so anyone can find it
on its own.
Basically, we detected 5% regression on XDP_DROP+mlx5 (currently, we
don't have data for any other XDP mode) in kernel-5.14 compared to
previous builds.

From tests history, I can see (most likely) the same improvement
on 6.10rc2 (from 15Mpps to 17-18Mpps), so I'd say 20% drop has been
(partially) fixed?

For earlier 6.10. kernels we don't have data due to [3] (there is regression on
XDP_DROP as well, but I believe it's turbo-boost issue, as I mentioned
in issue).
So if you want to run tests on 6.10. please see [3].

Summary XDP_DROP+mlx5@25G:
kernel       pps
<5.14        20.5M        baseline
>=5.14      19M           [0]
<6.4          19-20M      baseline for ELN kernels
>=6.4        15M           [4 and 5] (mentioned by Toke)
>=6.10      ???            [3]
>=6.10rc2 17M-18M


> It looks like this is known since March, was this ever reported to Nvidia back
> then? :/

Not sure if that's a question for me, I was told, filling an issue in
Bugzilla/Jira is where
our competences end. Who is supposed to report it to them?

> Given XDP is in the critical path for many in production, we should think about
> regular performance reporting for the different vendors for each released kernel,
> similar to here [0].

I think this might be the part of upstream kernel testing with LNST?
Maybe Jesper
knows more about that? Until then, I think, I can let you know about
new regressions we catch.

Thanks,
Sam.

[0] https://issues.redhat.com/browse/RHEL-24054
[1] https://koji.fedoraproject.org/koji/search?terms=kernel-%5Cd.*eln*&type=build&match=regexp
[2] https://koji.fedoraproject.org/koji/buildinfo?buildID=2469107
[3] https://bugzilla.redhat.com/show_bug.cgi?id=2282969
[4] https://bugzilla.redhat.com/show_bug.cgi?id=2270408
[5] https://issues.redhat.com/browse/RHEL-24054

Re: XDP Performance Regression in recent kernel versions

[Toke Høiland-Jørgensen]

Samuel Dobron <sdobron@redhat.com> writes:

>> It looks like this is known since March, was this ever reported to Nvidia back
>> then? :/
>
> Not sure if that's a question for me, I was told, filling an issue in
> Bugzilla/Jira is where
> our competences end. Who is supposed to report it to them?

I don't think we have a formal reporting procedure, but I was planning
to send this to the list, referencing the Bugzilla entry. Seems I
dropped the ball on that; sorry! :(

Can we set up a better reporting procedure for this going forward? A
mailing list, or just a name we can put in reports? Or something else?
Tariq, any preferences?

-Toke

Re: XDP Performance Regression in recent kernel versions

[Tariq Toukan]

On 24/06/2024 14:46, Toke Høiland-Jørgensen wrote:
> Samuel Dobron <sdobron@redhat.com> writes:
> 
>>> It looks like this is known since March, was this ever reported to Nvidia back
>>> then? :/
>>
>> Not sure if that's a question for me, I was told, filling an issue in
>> Bugzilla/Jira is where
>> our competences end. Who is supposed to report it to them?
> 
> I don't think we have a formal reporting procedure, but I was planning
> to send this to the list, referencing the Bugzilla entry. Seems I
> dropped the ball on that; sorry! :(
> 
> Can we set up a better reporting procedure for this going forward? A
> mailing list, or just a name we can put in reports? Or something else?
> Tariq, any preferences?
> 
> -Toke
> 

Hi,
Please add Dragos and me on XDP mailing list reports.

Regards,
Tariq

Re: XDP Performance Regression in recent kernel versions

[Tariq Toukan]

On 21/06/2024 15:35, Samuel Dobron wrote:
> Hey all,
> 
> Yeah, we do tests for ELN kernels [1] on a regular basis. Since
> ~January of this year.
> 
> As already mentioned, mlx5 is the only driver affected by this regression.
> Unfortunately, I think Jesper is actually hitting 2 regressions we noticed,
> the one already mentioned by Toke, another one [0] has been reported
> in early February.
> Btw. issue mentioned by Toke has been moved to Jira, see [5].
> 
> Not sure all of you are able to see the content of [0], Jira says it's
> RH-confidental.
> So, I am not sure how much I can share without being fired :D. Anyway,
> affected kernels have been released a while ago, so anyone can find it
> on its own.
> Basically, we detected 5% regression on XDP_DROP+mlx5 (currently, we
> don't have data for any other XDP mode) in kernel-5.14 compared to
> previous builds.
> 
>  From tests history, I can see (most likely) the same improvement
> on 6.10rc2 (from 15Mpps to 17-18Mpps), so I'd say 20% drop has been
> (partially) fixed?
> 
> For earlier 6.10. kernels we don't have data due to [3] (there is regression on
> XDP_DROP as well, but I believe it's turbo-boost issue, as I mentioned
> in issue).
> So if you want to run tests on 6.10. please see [3].
> 
> Summary XDP_DROP+mlx5@25G:
> kernel       pps
> <5.14        20.5M        baseline
>> =5.14      19M           [0]
> <6.4          19-20M      baseline for ELN kernels
>> =6.4        15M           [4 and 5] (mentioned by Toke)

+ @Dragos

That's about when we added several changes to the RX datapath.
Most relevant are:
- Fully removing the in-driver RX page-cache.
- Refactoring to support XDP multi-buffer.

We tested XDP performance before submission, I don't recall we noticed 
such a degradation.

I'll check with Dragos as he probably has these reports.

>> =6.10      ???            [3]
>> =6.10rc2 17M-18M
> 
> 
>> It looks like this is known since March, was this ever reported to Nvidia back
>> then? :/
> 
> Not sure if that's a question for me, I was told, filling an issue in
> Bugzilla/Jira is where
> our competences end. Who is supposed to report it to them?
> 
>> Given XDP is in the critical path for many in production, we should think about
>> regular performance reporting for the different vendors for each released kernel,
>> similar to here [0].
> 
> I think this might be the part of upstream kernel testing with LNST?
> Maybe Jesper
> knows more about that? Until then, I think, I can let you know about
> new regressions we catch.
> 
> Thanks,
> Sam.
> 
> [0] https://issues.redhat.com/browse/RHEL-24054
> [1] https://koji.fedoraproject.org/koji/search?terms=kernel-%5Cd.*eln*&type=build&match=regexp
> [2] https://koji.fedoraproject.org/koji/buildinfo?buildID=2469107
> [3] https://bugzilla.redhat.com/show_bug.cgi?id=2282969
> [4] https://bugzilla.redhat.com/show_bug.cgi?id=2270408
> [5] https://issues.redhat.com/browse/RHEL-24054
> 

Re: XDP Performance Regression in recent kernel versions

[Dragos Tatulea]

On Sun, 2024-06-30 at 14:43 +0300, Tariq Toukan wrote:
> 
> On 21/06/2024 15:35, Samuel Dobron wrote:
> > Hey all,
> > 
> > Yeah, we do tests for ELN kernels [1] on a regular basis. Since
> > ~January of this year.
> > 
> > As already mentioned, mlx5 is the only driver affected by this regression.
> > Unfortunately, I think Jesper is actually hitting 2 regressions we noticed,
> > the one already mentioned by Toke, another one [0] has been reported
> > in early February.
> > Btw. issue mentioned by Toke has been moved to Jira, see [5].
> > 
> > Not sure all of you are able to see the content of [0], Jira says it's
> > RH-confidental.
> > So, I am not sure how much I can share without being fired :D. Anyway,
> > affected kernels have been released a while ago, so anyone can find it
> > on its own.
> > Basically, we detected 5% regression on XDP_DROP+mlx5 (currently, we
> > don't have data for any other XDP mode) in kernel-5.14 compared to
> > previous builds.
> > 
> >  From tests history, I can see (most likely) the same improvement
> > on 6.10rc2 (from 15Mpps to 17-18Mpps), so I'd say 20% drop has been
> > (partially) fixed?
> > 
> > For earlier 6.10. kernels we don't have data due to [3] (there is regression on
> > XDP_DROP as well, but I believe it's turbo-boost issue, as I mentioned
> > in issue).
> > So if you want to run tests on 6.10. please see [3].
> > 
> > Summary XDP_DROP+mlx5@25G:
> > kernel       pps
> > <5.14        20.5M        baseline
> > > =5.14      19M           [0]
> > <6.4          19-20M      baseline for ELN kernels
> > > =6.4        15M           [4 and 5] (mentioned by Toke)
> 
> + @Dragos
> 
> That's about when we added several changes to the RX datapath.
> Most relevant are:
> - Fully removing the in-driver RX page-cache.
> - Refactoring to support XDP multi-buffer.
> 
> We tested XDP performance before submission, I don't recall we noticed 
> such a degradation.

Adding Carolina to post her analysis on this.

> 
> I'll check with Dragos as he probably has these reports.
> 
We only noticed a 6% degradation for XDP_XDROP.

https://lore.kernel.org/netdev/b6fcfa8b-c2b3-8a92-fb6e-0760d5f6f5ff@redhat.com/T/

> > > =6.10      ???            [3]
> > > =6.10rc2 17M-18M
> > 
> > 
> > > It looks like this is known since March, was this ever reported to Nvidia back
> > > then? :/
> > 
> > Not sure if that's a question for me, I was told, filling an issue in
> > Bugzilla/Jira is where
> > our competences end. Who is supposed to report it to them?
> > 
> > > Given XDP is in the critical path for many in production, we should think about
> > > regular performance reporting for the different vendors for each released kernel,
> > > similar to here [0].
> > 
> > I think this might be the part of upstream kernel testing with LNST?
> > Maybe Jesper
> > knows more about that? Until then, I think, I can let you know about
> > new regressions we catch.
> > 
> > Thanks,
> > Sam.
> > 
> > [0] https://issues.redhat.com/browse/RHEL-24054
> > [1] https://koji.fedoraproject.org/koji/search?terms=kernel-%5Cd.*eln*&type=build&match=regexp
> > [2] https://koji.fedoraproject.org/koji/buildinfo?buildID=2469107
> > [3] https://bugzilla.redhat.com/show_bug.cgi?id=2282969
> > [4] https://bugzilla.redhat.com/show_bug.cgi?id=2270408
> > [5] https://issues.redhat.com/browse/RHEL-24054
> > 

Re: XDP Performance Regression in recent kernel versions

[Samuel Dobron]

Hey,

Sorry for waiting.
I've started a discussion within our team, how to handle this since we
don't have reporting process defined. So it may take some time, I'll
let you know.

Thanks,
Sam.

On Sun, Jun 30, 2024 at 12:26 PM Tariq Toukan <tariqt@nvidia.com> wrote:
>
>
>
> On 24/06/2024 14:46, Toke Høiland-Jørgensen wrote:
> > Samuel Dobron <sdobron@redhat.com> writes:
> >
> >>> It looks like this is known since March, was this ever reported to Nvidia back
> >>> then? :/
> >>
> >> Not sure if that's a question for me, I was told, filling an issue in
> >> Bugzilla/Jira is where
> >> our competences end. Who is supposed to report it to them?
> >
> > I don't think we have a formal reporting procedure, but I was planning
> > to send this to the list, referencing the Bugzilla entry. Seems I
> > dropped the ball on that; sorry! :(
> >
> > Can we set up a better reporting procedure for this going forward? A
> > mailing list, or just a name we can put in reports? Or something else?
> > Tariq, any preferences?
> >
> > -Toke
> >
>
> Hi,
> Please add Dragos and me on XDP mailing list reports.
>
> Regards,
> Tariq
>

Re: XDP Performance Regression in recent kernel versions

[Carolina Jubran]

On 22/07/2024 12:26, Dragos Tatulea wrote:
> On Sun, 2024-06-30 at 14:43 +0300, Tariq Toukan wrote:
>>
>> On 21/06/2024 15:35, Samuel Dobron wrote:
>>> Hey all,
>>>
>>> Yeah, we do tests for ELN kernels [1] on a regular basis. Since
>>> ~January of this year.
>>>
>>> As already mentioned, mlx5 is the only driver affected by this regression.
>>> Unfortunately, I think Jesper is actually hitting 2 regressions we noticed,
>>> the one already mentioned by Toke, another one [0] has been reported
>>> in early February.
>>> Btw. issue mentioned by Toke has been moved to Jira, see [5].
>>>
>>> Not sure all of you are able to see the content of [0], Jira says it's
>>> RH-confidental.
>>> So, I am not sure how much I can share without being fired :D. Anyway,
>>> affected kernels have been released a while ago, so anyone can find it
>>> on its own.
>>> Basically, we detected 5% regression on XDP_DROP+mlx5 (currently, we
>>> don't have data for any other XDP mode) in kernel-5.14 compared to
>>> previous builds.
>>>
>>>   From tests history, I can see (most likely) the same improvement
>>> on 6.10rc2 (from 15Mpps to 17-18Mpps), so I'd say 20% drop has been
>>> (partially) fixed?
>>>
>>> For earlier 6.10. kernels we don't have data due to [3] (there is regression on
>>> XDP_DROP as well, but I believe it's turbo-boost issue, as I mentioned
>>> in issue).
>>> So if you want to run tests on 6.10. please see [3].
>>>
>>> Summary XDP_DROP+mlx5@25G:
>>> kernel       pps
>>> <5.14        20.5M        baseline
>>>> =5.14      19M           [0]
>>> <6.4          19-20M      baseline for ELN kernels
>>>> =6.4        15M           [4 and 5] (mentioned by Toke)
>>
>> + @Dragos
>>
>> That's about when we added several changes to the RX datapath.
>> Most relevant are:
>> - Fully removing the in-driver RX page-cache.
>> - Refactoring to support XDP multi-buffer.
>>
>> We tested XDP performance before submission, I don't recall we noticed
>> such a degradation.
> 
> Adding Carolina to post her analysis on this.

Hey everyone,

After investigating the issue, it seems the performance degradation is 
linked to the commit "x86/bugs: Report Intel retbleed vulnerability"
(6ad0ad2bf8a67).

This commit addresses the Intel retbleed vulnerability and introduces
mitigation measures that impact performance, especially the Spectre v2
mitigations.


Disabling these mitigations in the kernel arguments
(spectre_v2=off ibrs=off) resolved the degradation in my tests.

Could you try adding the mentioned parameters to your kernel arguments
and check if you still see the degradation?

Thank you,

Carolina.

> 
>>
>> I'll check with Dragos as he probably has these reports.
>>
> We only noticed a 6% degradation for XDP_XDROP.
> 
> https://lore.kernel.org/netdev/b6fcfa8b-c2b3-8a92-fb6e-0760d5f6f5ff@redhat.com/T/
> 
>>>> =6.10      ???            [3]
>>>> =6.10rc2 17M-18M
>>>
>>>
>>>> It looks like this is known since March, was this ever reported to Nvidia back
>>>> then? :/
>>>
>>> Not sure if that's a question for me, I was told, filling an issue in
>>> Bugzilla/Jira is where
>>> our competences end. Who is supposed to report it to them?
>>>
>>>> Given XDP is in the critical path for many in production, we should think about
>>>> regular performance reporting for the different vendors for each released kernel,
>>>> similar to here [0].
>>>
>>> I think this might be the part of upstream kernel testing with LNST?
>>> Maybe Jesper
>>> knows more about that? Until then, I think, I can let you know about
>>> new regressions we catch.
>>>
>>> Thanks,
>>> Sam.
>>>
>>> [0] https://issues.redhat.com/browse/RHEL-24054
>>> [1] https://koji.fedoraproject.org/koji/search?terms=kernel-%5Cd.*eln*&type=build&match=regexp
>>> [2] https://koji.fedoraproject.org/koji/buildinfo?buildID=2469107
>>> [3] https://bugzilla.redhat.com/show_bug.cgi?id=2282969
>>> [4] https://bugzilla.redhat.com/show_bug.cgi?id=2270408
>>> [5] https://issues.redhat.com/browse/RHEL-24054
>>>
> 

Re: XDP Performance Regression in recent kernel versions

[Toke Høiland-Jørgensen]

Carolina Jubran <cjubran@nvidia.com> writes:

> On 22/07/2024 12:26, Dragos Tatulea wrote:
>> On Sun, 2024-06-30 at 14:43 +0300, Tariq Toukan wrote:
>>>
>>> On 21/06/2024 15:35, Samuel Dobron wrote:
>>>> Hey all,
>>>>
>>>> Yeah, we do tests for ELN kernels [1] on a regular basis. Since
>>>> ~January of this year.
>>>>
>>>> As already mentioned, mlx5 is the only driver affected by this regression.
>>>> Unfortunately, I think Jesper is actually hitting 2 regressions we noticed,
>>>> the one already mentioned by Toke, another one [0] has been reported
>>>> in early February.
>>>> Btw. issue mentioned by Toke has been moved to Jira, see [5].
>>>>
>>>> Not sure all of you are able to see the content of [0], Jira says it's
>>>> RH-confidental.
>>>> So, I am not sure how much I can share without being fired :D. Anyway,
>>>> affected kernels have been released a while ago, so anyone can find it
>>>> on its own.
>>>> Basically, we detected 5% regression on XDP_DROP+mlx5 (currently, we
>>>> don't have data for any other XDP mode) in kernel-5.14 compared to
>>>> previous builds.
>>>>
>>>>   From tests history, I can see (most likely) the same improvement
>>>> on 6.10rc2 (from 15Mpps to 17-18Mpps), so I'd say 20% drop has been
>>>> (partially) fixed?
>>>>
>>>> For earlier 6.10. kernels we don't have data due to [3] (there is regression on
>>>> XDP_DROP as well, but I believe it's turbo-boost issue, as I mentioned
>>>> in issue).
>>>> So if you want to run tests on 6.10. please see [3].
>>>>
>>>> Summary XDP_DROP+mlx5@25G:
>>>> kernel       pps
>>>> <5.14        20.5M        baseline
>>>>> =5.14      19M           [0]
>>>> <6.4          19-20M      baseline for ELN kernels
>>>>> =6.4        15M           [4 and 5] (mentioned by Toke)
>>>
>>> + @Dragos
>>>
>>> That's about when we added several changes to the RX datapath.
>>> Most relevant are:
>>> - Fully removing the in-driver RX page-cache.
>>> - Refactoring to support XDP multi-buffer.
>>>
>>> We tested XDP performance before submission, I don't recall we noticed
>>> such a degradation.
>> 
>> Adding Carolina to post her analysis on this.
>
> Hey everyone,
>
> After investigating the issue, it seems the performance degradation is 
> linked to the commit "x86/bugs: Report Intel retbleed vulnerability"
> (6ad0ad2bf8a67).

Hmm, that commit is from June 2022, and according to Samuel's tests,
this issue was introduced sometime between commits b6dad5178cea and
40f71e7cd3c6 (both of which are dated in June 2023). Besides, if it was
a retbleed mitigation issue, that would affect other drivers as well,
no? Our testing only shows this regression on mlx5, not on the intel
drivers.


>>> I'll check with Dragos as he probably has these reports.
>>>
>> We only noticed a 6% degradation for XDP_XDROP.
>> 
>> https://lore.kernel.org/netdev/b6fcfa8b-c2b3-8a92-fb6e-0760d5f6f5ff@redhat.com/T/

That message mentions that "This will be handled in a different patch
series by adding support for multi-packet per page." - did that ever go
in?

-Toke

Re: XDP Performance Regression in recent kernel versions

[Samuel Dobron]

Confirming that this is just mlx5 issue, intel is fine.

I just did a quick test with disabled[0] Spectre v2 mitigations.
The performance remains the same, no difference at all.

Sam.

[0]:
$ cat /sys/devices/system/cpu/vulnerabilities/spectre_v2
Vulnerable; IBPB: disabled; STIBP: disabled; PBRSB-eIBRS: Vulnerable;
BHI: Vulnerable

On Wed, Jul 24, 2024 at 5:48 PM Toke Høiland-Jørgensen <toke@redhat.com> wrote:
>
> Carolina Jubran <cjubran@nvidia.com> writes:
>
> > On 22/07/2024 12:26, Dragos Tatulea wrote:
> >> On Sun, 2024-06-30 at 14:43 +0300, Tariq Toukan wrote:
> >>>
> >>> On 21/06/2024 15:35, Samuel Dobron wrote:
> >>>> Hey all,
> >>>>
> >>>> Yeah, we do tests for ELN kernels [1] on a regular basis. Since
> >>>> ~January of this year.
> >>>>
> >>>> As already mentioned, mlx5 is the only driver affected by this regression.
> >>>> Unfortunately, I think Jesper is actually hitting 2 regressions we noticed,
> >>>> the one already mentioned by Toke, another one [0] has been reported
> >>>> in early February.
> >>>> Btw. issue mentioned by Toke has been moved to Jira, see [5].
> >>>>
> >>>> Not sure all of you are able to see the content of [0], Jira says it's
> >>>> RH-confidental.
> >>>> So, I am not sure how much I can share without being fired :D. Anyway,
> >>>> affected kernels have been released a while ago, so anyone can find it
> >>>> on its own.
> >>>> Basically, we detected 5% regression on XDP_DROP+mlx5 (currently, we
> >>>> don't have data for any other XDP mode) in kernel-5.14 compared to
> >>>> previous builds.
> >>>>
> >>>>   From tests history, I can see (most likely) the same improvement
> >>>> on 6.10rc2 (from 15Mpps to 17-18Mpps), so I'd say 20% drop has been
> >>>> (partially) fixed?
> >>>>
> >>>> For earlier 6.10. kernels we don't have data due to [3] (there is regression on
> >>>> XDP_DROP as well, but I believe it's turbo-boost issue, as I mentioned
> >>>> in issue).
> >>>> So if you want to run tests on 6.10. please see [3].
> >>>>
> >>>> Summary XDP_DROP+mlx5@25G:
> >>>> kernel       pps
> >>>> <5.14        20.5M        baseline
> >>>>> =5.14      19M           [0]
> >>>> <6.4          19-20M      baseline for ELN kernels
> >>>>> =6.4        15M           [4 and 5] (mentioned by Toke)
> >>>
> >>> + @Dragos
> >>>
> >>> That's about when we added several changes to the RX datapath.
> >>> Most relevant are:
> >>> - Fully removing the in-driver RX page-cache.
> >>> - Refactoring to support XDP multi-buffer.
> >>>
> >>> We tested XDP performance before submission, I don't recall we noticed
> >>> such a degradation.
> >>
> >> Adding Carolina to post her analysis on this.
> >
> > Hey everyone,
> >
> > After investigating the issue, it seems the performance degradation is
> > linked to the commit "x86/bugs: Report Intel retbleed vulnerability"
> > (6ad0ad2bf8a67).
>
> Hmm, that commit is from June 2022, and according to Samuel's tests,
> this issue was introduced sometime between commits b6dad5178cea and
> 40f71e7cd3c6 (both of which are dated in June 2023). Besides, if it was
> a retbleed mitigation issue, that would affect other drivers as well,
> no? Our testing only shows this regression on mlx5, not on the intel
> drivers.
>
>
> >>> I'll check with Dragos as he probably has these reports.
> >>>
> >> We only noticed a 6% degradation for XDP_XDROP.
> >>
> >> https://lore.kernel.org/netdev/b6fcfa8b-c2b3-8a92-fb6e-0760d5f6f5ff@redhat.com/T/
>
> That message mentions that "This will be handled in a different patch
> series by adding support for multi-packet per page." - did that ever go
> in?
>
> -Toke
>

Re: XDP Performance Regression in recent kernel versions

[Dragos Tatulea]

Hi,

On Wed, 2024-07-24 at 17:36 +0200, Toke Høiland-Jørgensen wrote:
> Carolina Jubran <cjubran@nvidia.com> writes:
> 
> > On 22/07/2024 12:26, Dragos Tatulea wrote:
> > > On Sun, 2024-06-30 at 14:43 +0300, Tariq Toukan wrote:
> > > > 
> > > > On 21/06/2024 15:35, Samuel Dobron wrote:
> > > > > Hey all,
> > > > > 
> > > > > Yeah, we do tests for ELN kernels [1] on a regular basis. Since
> > > > > ~January of this year.
> > > > > 
> > > > > As already mentioned, mlx5 is the only driver affected by this regression.
> > > > > Unfortunately, I think Jesper is actually hitting 2 regressions we noticed,
> > > > > the one already mentioned by Toke, another one [0] has been reported
> > > > > in early February.
> > > > > Btw. issue mentioned by Toke has been moved to Jira, see [5].
> > > > > 
> > > > > Not sure all of you are able to see the content of [0], Jira says it's
> > > > > RH-confidental.
> > > > > So, I am not sure how much I can share without being fired :D. Anyway,
> > > > > affected kernels have been released a while ago, so anyone can find it
> > > > > on its own.
> > > > > Basically, we detected 5% regression on XDP_DROP+mlx5 (currently, we
> > > > > don't have data for any other XDP mode) in kernel-5.14 compared to
> > > > > previous builds.
> > > > > 
> > > > >   From tests history, I can see (most likely) the same improvement
> > > > > on 6.10rc2 (from 15Mpps to 17-18Mpps), so I'd say 20% drop has been
> > > > > (partially) fixed?
> > > > > 
> > > > > For earlier 6.10. kernels we don't have data due to [3] (there is regression on
> > > > > XDP_DROP as well, but I believe it's turbo-boost issue, as I mentioned
> > > > > in issue).
> > > > > So if you want to run tests on 6.10. please see [3].
> > > > > 
> > > > > Summary XDP_DROP+mlx5@25G:
> > > > > kernel       pps
> > > > > <5.14        20.5M        baseline
> > > > > > =5.14      19M           [0]
> > > > > <6.4          19-20M      baseline for ELN kernels
> > > > > > =6.4        15M           [4 and 5] (mentioned by Toke)
> > > > 
> > > > + @Dragos
> > > > 
> > > > That's about when we added several changes to the RX datapath.
> > > > Most relevant are:
> > > > - Fully removing the in-driver RX page-cache.
> > > > - Refactoring to support XDP multi-buffer.
> > > > 
> > > > We tested XDP performance before submission, I don't recall we noticed
> > > > such a degradation.
> > > 
> > > Adding Carolina to post her analysis on this.
> > 
> > Hey everyone,
> > 
> > After investigating the issue, it seems the performance degradation is 
> > linked to the commit "x86/bugs: Report Intel retbleed vulnerability"
> > (6ad0ad2bf8a67).
> 
> Hmm, that commit is from June 2022, [...]
> 
The results from the very first mail in this thread from Sebastiano were
showing a 30Mpps -> 21.3Mpps XDP_DROP regression between 5.15 and 6.2. This
is what Carolina was focused on. Furthermore, the results from Samuel don't show
this regression. Seems like the discussion is now focused on the 6.4 regression?

> [...] and according to Samuel's tests,
> this issue was introduced sometime between commits b6dad5178cea and
> 40f71e7cd3c6 (both of which are dated in June 2023).
> 
Thanks for the commit range (now I know how to decode ELN kernel versions :)).
Strangely this range doesn't have anything suspicious. I would have expected to
see the page_pool or the XDP multibuf changes would have shown up in this range.
But they are already present in the working version... Anyway, we'll keep on
looking.

>  Besides, if it was
> a retbleed mitigation issue, that would affect other drivers as well,
> no? Our testing only shows this regression on mlx5, not on the intel
> drivers.
> 
> 
> > > > I'll check with Dragos as he probably has these reports.
> > > > 
> > > We only noticed a 6% degradation for XDP_XDROP.
> > > 
> > > https://lore.kernel.org/netdev/b6fcfa8b-c2b3-8a92-fb6e-0760d5f6f5ff@redhat.com/T/
> 
> That message mentions that "This will be handled in a different patch
> series by adding support for multi-packet per page." - did that ever go
> in?
> 
Nope, no XDP multi-packet per page yet.

Thanks,
Dragos

Re: XDP Performance Regression in recent kernel versions

[Samuel Dobron]

Ah, sorry.
Yes, I was talking about 6.4 regression.

I double-checked that v5.15 regression and I don't see anything
that significant as Sebastiano. I ran a couple of tests for:
* kernel-5.10.0-0.rc6.90.eln105
* kernel-5.14.0-60.eln112
* kernel-5.15.0-0.rc7.53.eln113
* kernel-5.16.0-60.eln114
* kernel-6.11.0-0.rc0.20240724git786c8248dbd3.12.eln141

The results of XDP_DROP on receiving side (the one, that is dropping
packets) are more or less the same ~20.5Mpps (17.5Mpps on 6.11, but
that's due to 6.4 regression). CPU is bottleneck, so 100% cpu utilization
for all the kernels on both ends - generator and receiver. We use pktgen
as a generator, both generator and receiver machines use mlx5 NIC.

However, I noticed that between 5.10 and 5.14 there is 30Mpps->22Mpps
regression BUT at the GENERATOR side, CPU util remains the same
on both ends and amount of dropped packets on receiver side is
the same as well (since it's CPU bottlenecked). Other drivers seems
to be unaffected.

That's probably something unrelated to Sebastiano's regression,
but I believe it's worth to mention.

And so, no idea where Sebastiano's regression comes from. I can see,
he uses ConnectX-6, we don't have those, only ConnectX-5, cloud that
be the problem?

Thanks,
Sam.




On Fri, Jul 26, 2024 at 10:09 AM Dragos Tatulea <dtatulea@nvidia.com> wrote:
>
> Hi,
>
> On Wed, 2024-07-24 at 17:36 +0200, Toke Høiland-Jørgensen wrote:
> > Carolina Jubran <cjubran@nvidia.com> writes:
> >
> > > On 22/07/2024 12:26, Dragos Tatulea wrote:
> > > > On Sun, 2024-06-30 at 14:43 +0300, Tariq Toukan wrote:
> > > > >
> > > > > On 21/06/2024 15:35, Samuel Dobron wrote:
> > > > > > Hey all,
> > > > > >
> > > > > > Yeah, we do tests for ELN kernels [1] on a regular basis. Since
> > > > > > ~January of this year.
> > > > > >
> > > > > > As already mentioned, mlx5 is the only driver affected by this regression.
> > > > > > Unfortunately, I think Jesper is actually hitting 2 regressions we noticed,
> > > > > > the one already mentioned by Toke, another one [0] has been reported
> > > > > > in early February.
> > > > > > Btw. issue mentioned by Toke has been moved to Jira, see [5].
> > > > > >
> > > > > > Not sure all of you are able to see the content of [0], Jira says it's
> > > > > > RH-confidental.
> > > > > > So, I am not sure how much I can share without being fired :D. Anyway,
> > > > > > affected kernels have been released a while ago, so anyone can find it
> > > > > > on its own.
> > > > > > Basically, we detected 5% regression on XDP_DROP+mlx5 (currently, we
> > > > > > don't have data for any other XDP mode) in kernel-5.14 compared to
> > > > > > previous builds.
> > > > > >
> > > > > >   From tests history, I can see (most likely) the same improvement
> > > > > > on 6.10rc2 (from 15Mpps to 17-18Mpps), so I'd say 20% drop has been
> > > > > > (partially) fixed?
> > > > > >
> > > > > > For earlier 6.10. kernels we don't have data due to [3] (there is regression on
> > > > > > XDP_DROP as well, but I believe it's turbo-boost issue, as I mentioned
> > > > > > in issue).
> > > > > > So if you want to run tests on 6.10. please see [3].
> > > > > >
> > > > > > Summary XDP_DROP+mlx5@25G:
> > > > > > kernel       pps
> > > > > > <5.14        20.5M        baseline
> > > > > > > =5.14      19M           [0]
> > > > > > <6.4          19-20M      baseline for ELN kernels
> > > > > > > =6.4        15M           [4 and 5] (mentioned by Toke)
> > > > >
> > > > > + @Dragos
> > > > >
> > > > > That's about when we added several changes to the RX datapath.
> > > > > Most relevant are:
> > > > > - Fully removing the in-driver RX page-cache.
> > > > > - Refactoring to support XDP multi-buffer.
> > > > >
> > > > > We tested XDP performance before submission, I don't recall we noticed
> > > > > such a degradation.
> > > >
> > > > Adding Carolina to post her analysis on this.
> > >
> > > Hey everyone,
> > >
> > > After investigating the issue, it seems the performance degradation is
> > > linked to the commit "x86/bugs: Report Intel retbleed vulnerability"
> > > (6ad0ad2bf8a67).
> >
> > Hmm, that commit is from June 2022, [...]
> >
> The results from the very first mail in this thread from Sebastiano were
> showing a 30Mpps -> 21.3Mpps XDP_DROP regression between 5.15 and 6.2. This
> is what Carolina was focused on. Furthermore, the results from Samuel don't show
> this regression. Seems like the discussion is now focused on the 6.4 regression?
>
> > [...] and according to Samuel's tests,
> > this issue was introduced sometime between commits b6dad5178cea and
> > 40f71e7cd3c6 (both of which are dated in June 2023).
> >
> Thanks for the commit range (now I know how to decode ELN kernel versions :)).
> Strangely this range doesn't have anything suspicious. I would have expected to
> see the page_pool or the XDP multibuf changes would have shown up in this range.
> But they are already present in the working version... Anyway, we'll keep on
> looking.
>
> >  Besides, if it was
> > a retbleed mitigation issue, that would affect other drivers as well,
> > no? Our testing only shows this regression on mlx5, not on the intel
> > drivers.
> >
> >
> > > > > I'll check with Dragos as he probably has these reports.
> > > > >
> > > > We only noticed a 6% degradation for XDP_XDROP.
> > > >
> > > > https://lore.kernel.org/netdev/b6fcfa8b-c2b3-8a92-fb6e-0760d5f6f5ff@redhat.com/T/
> >
> > That message mentions that "This will be handled in a different patch
> > series by adding support for multi-packet per page." - did that ever go
> > in?
> >
> Nope, no XDP multi-packet per page yet.
>
> Thanks,
> Dragos

Re: XDP Performance Regression in recent kernel versions

[Samuel Dobron]

> Could you try adding the mentioned parameters to your kernel arguments
> and check if you still see the degradation? 

Hey,
So i tried multiple kernels around v5.15 as well as couple of previous
v6.xx and there is no difference with spectre v2 mitigations enabled
or disabled.

No difference on other drivers as well.


Sam.

Re: XDP Performance Regression in recent kernel versions

[Samuel Dobron]

Hey all,

We recently enabled tests for XDP TX, so I was able to test
xdp tx as well.

XDP_DROP performance regression is the same as I reported
a while ago. There is about 20% regression in
kernel-6.4.0-0.rc6.20230616git40f71e7cd3c6.50.eln126 (baseline)
compared to previous kernel
kernel-6.4.0-0.rc6.20230614gitb6dad5178cea.49.eln126 (broken).
We don't see such regression for other drivers.

The regression was partially fixed somewhere between eln126 and
kernel-6.10.0-0.rc2.20240606git2df0193e62cf.27.eln137 (partially
fixed) and the performance since then is -7 to -15% compared to
baseline. So, nothing new.

XDP_TX is however, more interesting.
When comparing baseline with broken kernel there is 20 - 25%
performance drop (cpu utilizations remains the same) on mlx driver.
There is also 10% drop on other drivers as well. HOWEVER, it got
fixed somewhere between broken and partially fixed kernel. On most
recent kernels, we don't see that regressions on other drivers. But
2-10% (depends if using dpa/load-bytes) regression remains on mlx5.

The numbers look a bit similar to regression with enabled spectre/meltdown
mitigations but based on my experiments, there is no difference with
enabled/disabled mitigations.

Hope this will help,
Sam.

On Tue, Jul 30, 2024 at 1:04 PM Samuel Dobron <sdobron@redhat.com> wrote:
>
> > Could you try adding the mentioned parameters to your kernel arguments
> > and check if you still see the degradation?
>
> Hey,
> So i tried multiple kernels around v5.15 as well as couple of previous
> v6.xx and there is no difference with spectre v2 mitigations enabled
> or disabled.
>
> No difference on other drivers as well.
>
>
> Sam.
>

Re: XDP Performance Regression in recent kernel versions

[Carolina Jubran]

Hello,

Thank you Sam for the detailed information.

I have identified the specific kernel configuration change responsible 
for the degradation between kernel versions 
6.4.0-0.rc6.20230614gitb6dad5178cea.49.eln126 and 
6.4.0-0.rc6.20230616git40f71e7cd3c6.50.eln126. The introduction of the 
CONFIG_INIT_STACK_ALL_ZERO setting in the latter version has led to a 
noticeable performance impact.

I am currently investigating why this change specifically affects mlx5.

Thanks,

Carolina

On 11/12/2024 15:20, Samuel Dobron wrote:
> Hey all,
> 
> We recently enabled tests for XDP TX, so I was able to test
> xdp tx as well.
> 
> XDP_DROP performance regression is the same as I reported
> a while ago. There is about 20% regression in
> kernel-6.4.0-0.rc6.20230616git40f71e7cd3c6.50.eln126 (baseline)
> compared to previous kernel
> kernel-6.4.0-0.rc6.20230614gitb6dad5178cea.49.eln126 (broken).
> We don't see such regression for other drivers.
> 
> The regression was partially fixed somewhere between eln126 and
> kernel-6.10.0-0.rc2.20240606git2df0193e62cf.27.eln137 (partially
> fixed) and the performance since then is -7 to -15% compared to
> baseline. So, nothing new.
> 
> XDP_TX is however, more interesting.
> When comparing baseline with broken kernel there is 20 - 25%
> performance drop (cpu utilizations remains the same) on mlx driver.
> There is also 10% drop on other drivers as well. HOWEVER, it got
> fixed somewhere between broken and partially fixed kernel. On most
> recent kernels, we don't see that regressions on other drivers. But
> 2-10% (depends if using dpa/load-bytes) regression remains on mlx5.
> 
> The numbers look a bit similar to regression with enabled spectre/meltdown
> mitigations but based on my experiments, there is no difference with
> enabled/disabled mitigations.
> 
> Hope this will help,
> Sam.
> 
> On Tue, Jul 30, 2024 at 1:04 PM Samuel Dobron <sdobron@redhat.com> wrote:
>>
>>> Could you try adding the mentioned parameters to your kernel arguments
>>> and check if you still see the degradation?
>>
>> Hey,
>> So i tried multiple kernels around v5.15 as well as couple of previous
>> v6.xx and there is no difference with spectre v2 mitigations enabled
>> or disabled.
>>
>> No difference on other drivers as well.
>>
>>
>> Sam.
>>
>