netdev.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [PATCH v3 net-next 00/15] ipv6: No RTNL for IPv6 routing table.
@ 2025-04-18  0:03 Kuniyuki Iwashima
  2025-04-18  0:03 ` [PATCH v3 net-next 01/15] ipv6: Validate RTA_GATEWAY of RTA_MULTIPATH in rtm_to_fib6_config() Kuniyuki Iwashima
                   ` (15 more replies)
  0 siblings, 16 replies; 23+ messages in thread
From: Kuniyuki Iwashima @ 2025-04-18  0:03 UTC (permalink / raw)
  To: David S. Miller, David Ahern, Eric Dumazet, Jakub Kicinski,
	Paolo Abeni
  Cc: Simon Horman, Kuniyuki Iwashima, Kuniyuki Iwashima, netdev

IPv6 routing tables are protected by each table's lock and work in
the interrupt context, which means we basically don't need RTNL to
modify an IPv6 routing table itself.

Currently, the control paths require RTNL because we may need to
perform device and nexthop lookups; we must prevent dev/nexthop from
going away from the netns.

This, however, can be achieved by RCU as well.

If we are in the RCU critical section while adding an IPv6 route,
synchronize_net() in __dev_change_net_namespace() and
unregister_netdevice_many_notify() guarantee that the dev will not be
moved to another netns or removed.

Also, nexthop is guaranteed not to be freed during the RCU grace period.

If we care about a race between nexthop removal and IPv6 route addition,
we can get rid of RTNL from the control paths.

Patch 1 moves a validation for RTA_MULTIPATH earlier.
Patch 2 removes RTNL for SIOCDELRT and RTM_DELROUTE.
Patch 3 ~ 11 moves validation and memory allocation earlier.
Patch 12 prevents a race between two requests for the same table.
Patch 13 & 14 prevents the nexthop race mentioned above.
Patch 15 removes RTNL for SIOCADDRT and RTM_NEWROUTE.


Test:

The script [0] lets each CPU-X create 100000 routes on table-X in a
batch.

On c7a.metal-48xl EC2 instance with 192 CPUs,

without this series:

  $ sudo ./route_test.sh
  start adding routes
  added 19200000 routes (100000 routes * 192 tables).
  total routes: 19200006
  Time elapsed: 191577 milliseconds.

with this series:

  $ sudo ./route_test.sh
  start adding routes
  added 19200000 routes (100000 routes * 192 tables).
  total routes: 19200006
  Time elapsed: 62854 milliseconds.

I changed the number of routes (1000 ~ 100000 per CPU/table) and
consistently saw it finish 3x faster with this series.


[0]
#!/bin/bash

mkdir tmp

NS="test"
ip netns add $NS
ip -n $NS link add veth0 type veth peer veth1
ip -n $NS link set veth0 up
ip -n $NS link set veth1 up

TABLES=()
for i in $(seq $(nproc)); do
    TABLES+=("$i")
done

ROUTES=()
for i in {1..100}; do
    for j in {1..1000}; do
	ROUTES+=("2001:$i:$j::/64")
    done
done

for TABLE in "${TABLES[@]}"; do
    (
	FILE="./tmp/batch-table-$TABLE.txt"
	> $FILE
	for ROUTE in "${ROUTES[@]}"; do
            echo "route add $ROUTE dev veth0 table $TABLE" >> $FILE
	done
    ) &
done

wait

echo "start adding routes"

START_TIME=$(date +%s%3N)
for TABLE in "${TABLES[@]}"; do
    ip -n $NS -6 -batch "./tmp/batch-table-$TABLE.txt" &
done

wait
END_TIME=$(date +%s%3N)
ELAPSED_TIME=$((END_TIME - START_TIME))

echo "added $((${#ROUTES[@]} * ${#TABLES[@]})) routes (${#ROUTES[@]} routes * ${#TABLES[@]} tables)."
echo "total routes: $(ip -n $NS -6 route show table all | wc -l)"  # Just for debug
echo "Time elapsed: ${ELAPSED_TIME} milliseconds."

ip netns del $NS
rm -fr ./tmp/


Changes:
  v3:
    * Add patch 10
    * Patch 2
      * Add a note that fib6_get_table() does not require RCU
    * Patch 4
      * Explain the checks do not require RCU.
    * Patch 6
      * Update changelog s/everything as possible/as much as possible/
    * Patch 7
      * Explain alloc_percpu_gfp() is still needed when called via ipv6_stub.
    * Patch 14
      * Bundle critical section for rt->nh as fib6_add_rt2node_nh()
    * Patch 15
      * Add a note about lwtunnel_valid_encap_type()

  v2 (RESEND) : https://lore.kernel.org/netdev/20250414181516.28391-1-kuniyu@amazon.com/

  v2: https://lore.kernel.org/netdev/20250409011243.26195-1-kuniyu@amazon.com/
    * Add Patch 12
    * Patch 2
      * Call __ip6_del_rt() under RCU

  v1: https://lore.kernel.org/netdev/20250321040131.21057-1-kuniyu@amazon.com/


Kuniyuki Iwashima (15):
  ipv6: Validate RTA_GATEWAY of RTA_MULTIPATH in rtm_to_fib6_config().
  ipv6: Get rid of RTNL for SIOCDELRT and RTM_DELROUTE.
  ipv6: Move some validation from ip6_route_info_create() to
    rtm_to_fib6_config().
  ipv6: Check GATEWAY in rtm_to_fib6_multipath_config().
  ipv6: Move nexthop_find_by_id() after fib6_info_alloc().
  ipv6: Split ip6_route_info_create().
  ipv6: Preallocate rt->fib6_nh->rt6i_pcpu in ip6_route_info_create().
  ipv6: Preallocate nhc_pcpu_rth_output in ip6_route_info_create().
  ipv6: Don't pass net to ip6_route_info_append().
  ipv6: Rename rt6_nh.next to rt6_nh.list.
  ipv6: Factorise ip6_route_multipath_add().
  ipv6: Protect fib6_link_table() with spinlock.
  ipv6: Defer fib6_purge_rt() in fib6_add_rt2node() to fib6_add().
  ipv6: Protect nh->f6i_list with spinlock and flag.
  ipv6: Get rid of RTNL for SIOCADDRT and RTM_NEWROUTE.

 include/net/ip6_fib.h    |   1 +
 include/net/netns/ipv6.h |   1 +
 include/net/nexthop.h    |   2 +
 net/ipv4/fib_semantics.c |  10 +-
 net/ipv4/nexthop.c       |  22 +-
 net/ipv6/ip6_fib.c       |  84 ++++--
 net/ipv6/route.c         | 577 ++++++++++++++++++++++++---------------
 7 files changed, 452 insertions(+), 245 deletions(-)

-- 
2.49.0


^ permalink raw reply	[flat|nested] 23+ messages in thread
end of thread, other threads:[~2025-05-04 20:11 UTC | newest]

Thread overview: 23+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-04-18  0:03 [PATCH v3 net-next 00/15] ipv6: No RTNL for IPv6 routing table Kuniyuki Iwashima
2025-04-18  0:03 ` [PATCH v3 net-next 01/15] ipv6: Validate RTA_GATEWAY of RTA_MULTIPATH in rtm_to_fib6_config() Kuniyuki Iwashima
2025-04-18  0:03 ` [PATCH v3 net-next 02/15] ipv6: Get rid of RTNL for SIOCDELRT and RTM_DELROUTE Kuniyuki Iwashima
2025-04-18  0:03 ` [PATCH v3 net-next 03/15] ipv6: Move some validation from ip6_route_info_create() to rtm_to_fib6_config() Kuniyuki Iwashima
2025-04-29  0:24   ` Lai, Yi
2025-04-29  1:20     ` Kuniyuki Iwashima
2025-04-18  0:03 ` [PATCH v3 net-next 04/15] ipv6: Check GATEWAY in rtm_to_fib6_multipath_config() Kuniyuki Iwashima
2025-04-18  0:03 ` [PATCH v3 net-next 05/15] ipv6: Move nexthop_find_by_id() after fib6_info_alloc() Kuniyuki Iwashima
2025-04-18  0:03 ` [PATCH v3 net-next 06/15] ipv6: Split ip6_route_info_create() Kuniyuki Iwashima
2025-04-18  0:03 ` [PATCH v3 net-next 07/15] ipv6: Preallocate rt->fib6_nh->rt6i_pcpu in ip6_route_info_create() Kuniyuki Iwashima
2025-04-18  0:03 ` [PATCH v3 net-next 08/15] ipv6: Preallocate nhc_pcpu_rth_output " Kuniyuki Iwashima
2025-04-18  0:03 ` [PATCH v3 net-next 09/15] ipv6: Don't pass net to ip6_route_info_append() Kuniyuki Iwashima
2025-04-18  0:03 ` [PATCH v3 net-next 10/15] ipv6: Rename rt6_nh.next to rt6_nh.list Kuniyuki Iwashima
2025-04-18  0:03 ` [PATCH v3 net-next 11/15] ipv6: Factorise ip6_route_multipath_add() Kuniyuki Iwashima
2025-04-18  0:03 ` [PATCH v3 net-next 12/15] ipv6: Protect fib6_link_table() with spinlock Kuniyuki Iwashima
2025-04-18  0:03 ` [PATCH v3 net-next 13/15] ipv6: Defer fib6_purge_rt() in fib6_add_rt2node() to fib6_add() Kuniyuki Iwashima
2025-04-18  0:03 ` [PATCH v3 net-next 14/15] ipv6: Protect nh->f6i_list with spinlock and flag Kuniyuki Iwashima
2025-04-18  0:03 ` [PATCH v3 net-next 15/15] ipv6: Get rid of RTNL for SIOCADDRT and RTM_NEWROUTE Kuniyuki Iwashima
2025-05-04  9:16   ` Eric Dumazet
2025-05-04 17:20     ` Kuniyuki Iwashima
2025-05-04 19:34       ` Eric Dumazet
2025-05-04 20:11         ` Kuniyuki Iwashima
2025-04-24  7:50 ` [PATCH v3 net-next 00/15] ipv6: No RTNL for IPv6 routing table patchwork-bot+netdevbpf

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).