* [PATCH net 1/3] tcp: move tp->scaling_ratio to tcp_sock_read_txrx group
2024-02-08 14:43 [PATCH net 0/3] net: more three misplaced fields Eric Dumazet
@ 2024-02-08 14:43 ` Eric Dumazet
2024-02-09 17:15 ` Simon Horman
2024-02-08 14:43 ` [PATCH net 2/3] tcp: move tp->tcp_usec_ts " Eric Dumazet
` (2 subsequent siblings)
3 siblings, 1 reply; 8+ messages in thread
From: Eric Dumazet @ 2024-02-08 14:43 UTC (permalink / raw)
To: David S . Miller, Jakub Kicinski, Paolo Abeni
Cc: netdev, Naman Gulati, Coco Li, Wei Wang, eric.dumazet,
Eric Dumazet
tp->scaling_ratio is a read mostly field, used in rx and tx fast paths.
Fixes: d5fed5addb2b ("tcp: reorganize tcp_sock fast path variables")
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Coco Li <lixiaoyan@google.com>
Cc: Wei Wang <weiwan@google.com>
---
Documentation/networking/net_cachelines/tcp_sock.rst | 2 +-
include/linux/tcp.h | 2 +-
net/ipv4/tcp.c | 3 ++-
3 files changed, 4 insertions(+), 3 deletions(-)
diff --git a/Documentation/networking/net_cachelines/tcp_sock.rst b/Documentation/networking/net_cachelines/tcp_sock.rst
index 97d7a5c8e01c02658c7f445ed92a2d1f7cc61d31..80391229147971005571f9a9a4fcdcf4e50089da 100644
--- a/Documentation/networking/net_cachelines/tcp_sock.rst
+++ b/Documentation/networking/net_cachelines/tcp_sock.rst
@@ -38,7 +38,7 @@ u32 max_window read_mostly -
u32 mss_cache read_mostly read_mostly tcp_rate_check_app_limited,tcp_current_mss,tcp_sync_mss,tcp_sndbuf_expand,tcp_tso_should_defer(tx);tcp_update_pacing_rate,tcp_clean_rtx_queue(rx)
u32 window_clamp read_mostly read_write tcp_rcv_space_adjust,__tcp_select_window
u32 rcv_ssthresh read_mostly - __tcp_select_window
-u82 scaling_ratio
+u8 scaling_ratio read_mostly read_mostly tcp_win_from_space
struct tcp_rack
u16 advmss - read_mostly tcp_rcv_space_adjust
u8 compressed_ack
diff --git a/include/linux/tcp.h b/include/linux/tcp.h
index 89b290d8c8dc9f115df7a295bc8d2512698db169..168f5dca66096589e7a213ffd6742a51fe43b4a9 100644
--- a/include/linux/tcp.h
+++ b/include/linux/tcp.h
@@ -221,6 +221,7 @@ struct tcp_sock {
u32 lost_out; /* Lost packets */
u32 sacked_out; /* SACK'd packets */
u16 tcp_header_len; /* Bytes of tcp header to send */
+ u8 scaling_ratio; /* see tcp_win_from_space() */
u8 chrono_type : 2, /* current chronograph type */
repair : 1,
is_sack_reneg:1, /* in recovery from loss with SACK reneg? */
@@ -352,7 +353,6 @@ struct tcp_sock {
u32 compressed_ack_rcv_nxt;
struct list_head tsq_node; /* anchor in tsq_tasklet.head list */
- u8 scaling_ratio; /* see tcp_win_from_space() */
/* Information of the most recently (s)acked skb */
struct tcp_rack {
u64 mstamp; /* (Re)sent time of the skb */
diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c
index 7e2481b9eae1b791e1ec65f39efa41837a9fcbd3..c82dc42f57c65df112f79080ff407cd98d11ce68 100644
--- a/net/ipv4/tcp.c
+++ b/net/ipv4/tcp.c
@@ -4615,7 +4615,8 @@ static void __init tcp_struct_check(void)
CACHELINE_ASSERT_GROUP_MEMBER(struct tcp_sock, tcp_sock_read_txrx, prr_out);
CACHELINE_ASSERT_GROUP_MEMBER(struct tcp_sock, tcp_sock_read_txrx, lost_out);
CACHELINE_ASSERT_GROUP_MEMBER(struct tcp_sock, tcp_sock_read_txrx, sacked_out);
- CACHELINE_ASSERT_GROUP_SIZE(struct tcp_sock, tcp_sock_read_txrx, 31);
+ CACHELINE_ASSERT_GROUP_MEMBER(struct tcp_sock, tcp_sock_read_txrx, scaling_ratio);
+ CACHELINE_ASSERT_GROUP_SIZE(struct tcp_sock, tcp_sock_read_txrx, 32);
/* RX read-mostly hotpath cache lines */
CACHELINE_ASSERT_GROUP_MEMBER(struct tcp_sock, tcp_sock_read_rx, copied_seq);
--
2.43.0.594.gd9cf4e227d-goog
^ permalink raw reply related [flat|nested] 8+ messages in thread* Re: [PATCH net 1/3] tcp: move tp->scaling_ratio to tcp_sock_read_txrx group
2024-02-08 14:43 ` [PATCH net 1/3] tcp: move tp->scaling_ratio to tcp_sock_read_txrx group Eric Dumazet
@ 2024-02-09 17:15 ` Simon Horman
0 siblings, 0 replies; 8+ messages in thread
From: Simon Horman @ 2024-02-09 17:15 UTC (permalink / raw)
To: Eric Dumazet
Cc: David S . Miller, Jakub Kicinski, Paolo Abeni, netdev,
Naman Gulati, Coco Li, Wei Wang, eric.dumazet
On Thu, Feb 08, 2024 at 02:43:21PM +0000, Eric Dumazet wrote:
> tp->scaling_ratio is a read mostly field, used in rx and tx fast paths.
>
> Fixes: d5fed5addb2b ("tcp: reorganize tcp_sock fast path variables")
> Signed-off-by: Eric Dumazet <edumazet@google.com>
> Cc: Coco Li <lixiaoyan@google.com>
> Cc: Wei Wang <weiwan@google.com>
Reviewed-by: Simon Horman <horms@kernel.org>
^ permalink raw reply [flat|nested] 8+ messages in thread
* [PATCH net 2/3] tcp: move tp->tcp_usec_ts to tcp_sock_read_txrx group
2024-02-08 14:43 [PATCH net 0/3] net: more three misplaced fields Eric Dumazet
2024-02-08 14:43 ` [PATCH net 1/3] tcp: move tp->scaling_ratio to tcp_sock_read_txrx group Eric Dumazet
@ 2024-02-08 14:43 ` Eric Dumazet
2024-02-09 17:15 ` Simon Horman
2024-02-08 14:43 ` [PATCH net 3/3] net-device: move lstats in net_device_read_txrx Eric Dumazet
2024-02-12 10:00 ` [PATCH net 0/3] net: more three misplaced fields patchwork-bot+netdevbpf
3 siblings, 1 reply; 8+ messages in thread
From: Eric Dumazet @ 2024-02-08 14:43 UTC (permalink / raw)
To: David S . Miller, Jakub Kicinski, Paolo Abeni
Cc: netdev, Naman Gulati, Coco Li, Wei Wang, eric.dumazet,
Eric Dumazet
tp->tcp_usec_ts is a read mostly field, used in rx and tx fast paths.
Fixes: d5fed5addb2b ("tcp: reorganize tcp_sock fast path variables")
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Coco Li <lixiaoyan@google.com>
Cc: Wei Wang <weiwan@google.com>
---
Documentation/networking/net_cachelines/tcp_sock.rst | 2 +-
include/linux/tcp.h | 4 ++--
2 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/Documentation/networking/net_cachelines/tcp_sock.rst b/Documentation/networking/net_cachelines/tcp_sock.rst
index 80391229147971005571f9a9a4fcdcf4e50089da..1c154cbd18487e385c8ae7a1e39d3b5f5ab086a2 100644
--- a/Documentation/networking/net_cachelines/tcp_sock.rst
+++ b/Documentation/networking/net_cachelines/tcp_sock.rst
@@ -44,7 +44,7 @@ u16 advmss - read_m
u8 compressed_ack
u8:2 dup_ack_counter
u8:1 tlp_retrans
-u8:1 tcp_usec_ts
+u8:1 tcp_usec_ts read_mostly read_mostly
u32 chrono_start read_write - tcp_chrono_start/stop(tcp_write_xmit,tcp_cwnd_validate,tcp_send_syn_data)
u32[3] chrono_stat read_write - tcp_chrono_start/stop(tcp_write_xmit,tcp_cwnd_validate,tcp_send_syn_data)
u8:2 chrono_type read_write - tcp_chrono_start/stop(tcp_write_xmit,tcp_cwnd_validate,tcp_send_syn_data)
diff --git a/include/linux/tcp.h b/include/linux/tcp.h
index 168f5dca66096589e7a213ffd6742a51fe43b4a9..a1c47a6d69b0efd7e62765fbd873c848da22aaec 100644
--- a/include/linux/tcp.h
+++ b/include/linux/tcp.h
@@ -224,6 +224,7 @@ struct tcp_sock {
u8 scaling_ratio; /* see tcp_win_from_space() */
u8 chrono_type : 2, /* current chronograph type */
repair : 1,
+ tcp_usec_ts : 1, /* TSval values in usec */
is_sack_reneg:1, /* in recovery from loss with SACK reneg? */
is_cwnd_limited:1;/* forward progress limited by snd_cwnd? */
__cacheline_group_end(tcp_sock_read_txrx);
@@ -368,8 +369,7 @@ struct tcp_sock {
u8 compressed_ack;
u8 dup_ack_counter:2,
tlp_retrans:1, /* TLP is a retransmission */
- tcp_usec_ts:1, /* TSval values in usec */
- unused:4;
+ unused:5;
u8 thin_lto : 1,/* Use linear timeouts for thin streams */
recvmsg_inq : 1,/* Indicate # of bytes in queue upon recvmsg */
fastopen_connect:1, /* FASTOPEN_CONNECT sockopt */
--
2.43.0.594.gd9cf4e227d-goog
^ permalink raw reply related [flat|nested] 8+ messages in thread* Re: [PATCH net 2/3] tcp: move tp->tcp_usec_ts to tcp_sock_read_txrx group
2024-02-08 14:43 ` [PATCH net 2/3] tcp: move tp->tcp_usec_ts " Eric Dumazet
@ 2024-02-09 17:15 ` Simon Horman
0 siblings, 0 replies; 8+ messages in thread
From: Simon Horman @ 2024-02-09 17:15 UTC (permalink / raw)
To: Eric Dumazet
Cc: David S . Miller, Jakub Kicinski, Paolo Abeni, netdev,
Naman Gulati, Coco Li, Wei Wang, eric.dumazet
On Thu, Feb 08, 2024 at 02:43:22PM +0000, Eric Dumazet wrote:
> tp->tcp_usec_ts is a read mostly field, used in rx and tx fast paths.
>
> Fixes: d5fed5addb2b ("tcp: reorganize tcp_sock fast path variables")
> Signed-off-by: Eric Dumazet <edumazet@google.com>
> Cc: Coco Li <lixiaoyan@google.com>
> Cc: Wei Wang <weiwan@google.com>
Reviewed-by: Simon Horman <horms@kernel.org>
^ permalink raw reply [flat|nested] 8+ messages in thread
* [PATCH net 3/3] net-device: move lstats in net_device_read_txrx
2024-02-08 14:43 [PATCH net 0/3] net: more three misplaced fields Eric Dumazet
2024-02-08 14:43 ` [PATCH net 1/3] tcp: move tp->scaling_ratio to tcp_sock_read_txrx group Eric Dumazet
2024-02-08 14:43 ` [PATCH net 2/3] tcp: move tp->tcp_usec_ts " Eric Dumazet
@ 2024-02-08 14:43 ` Eric Dumazet
2024-02-09 17:15 ` Simon Horman
2024-02-12 10:00 ` [PATCH net 0/3] net: more three misplaced fields patchwork-bot+netdevbpf
3 siblings, 1 reply; 8+ messages in thread
From: Eric Dumazet @ 2024-02-08 14:43 UTC (permalink / raw)
To: David S . Miller, Jakub Kicinski, Paolo Abeni
Cc: netdev, Naman Gulati, Coco Li, Wei Wang, eric.dumazet,
Eric Dumazet, Simon Horman
dev->lstats is notably used from loopback ndo_start_xmit()
and other virtual drivers.
Per cpu stats updates are dirtying per-cpu data,
but the pointer itself is read-only.
Fixes: 43a71cd66b9c ("net-device: reorganize net_device fast path variables")
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Coco Li <lixiaoyan@google.com>
Cc: Simon Horman <horms@kernel.org>
---
Documentation/networking/net_cachelines/net_device.rst | 4 ++--
include/linux/netdevice.h | 10 +++++-----
net/core/dev.c | 3 ++-
3 files changed, 9 insertions(+), 8 deletions(-)
diff --git a/Documentation/networking/net_cachelines/net_device.rst b/Documentation/networking/net_cachelines/net_device.rst
index e75a53593bb9606f1c0595d8f7227881ec932b9c..dceb49d56a91158232543e920c7ed23bed74106e 100644
--- a/Documentation/networking/net_cachelines/net_device.rst
+++ b/Documentation/networking/net_cachelines/net_device.rst
@@ -136,8 +136,8 @@ struct_netpoll_info* npinfo -
possible_net_t nd_net - read_mostly (dev_net)napi_busy_loop,tcp_v(4/6)_rcv,ip(v6)_rcv,ip(6)_input,ip(6)_input_finish
void* ml_priv
enum_netdev_ml_priv_type ml_priv_type
-struct_pcpu_lstats__percpu* lstats
-struct_pcpu_sw_netstats__percpu* tstats
+struct_pcpu_lstats__percpu* lstats read_mostly dev_lstats_add()
+struct_pcpu_sw_netstats__percpu* tstats read_mostly dev_sw_netstats_tx_add()
struct_pcpu_dstats__percpu* dstats
struct_garp_port* garp_port
struct_mrp_port* mrp_port
diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
index 118c40258d07b787adf518e576e75545e4bae846..ef7bfbb9849733fa7f1f097ba53a36a68cc3384b 100644
--- a/include/linux/netdevice.h
+++ b/include/linux/netdevice.h
@@ -2141,6 +2141,11 @@ struct net_device {
/* TXRX read-mostly hotpath */
__cacheline_group_begin(net_device_read_txrx);
+ union {
+ struct pcpu_lstats __percpu *lstats;
+ struct pcpu_sw_netstats __percpu *tstats;
+ struct pcpu_dstats __percpu *dstats;
+ };
unsigned int flags;
unsigned short hard_header_len;
netdev_features_t features;
@@ -2395,11 +2400,6 @@ struct net_device {
enum netdev_ml_priv_type ml_priv_type;
enum netdev_stat_type pcpu_stat_type:8;
- union {
- struct pcpu_lstats __percpu *lstats;
- struct pcpu_sw_netstats __percpu *tstats;
- struct pcpu_dstats __percpu *dstats;
- };
#if IS_ENABLED(CONFIG_GARP)
struct garp_port __rcu *garp_port;
diff --git a/net/core/dev.c b/net/core/dev.c
index cb2dab0feee0abe758479a7a001342bf6613df08..9bb792cecc16f07449a91e4ca96357600d7453f9 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -11652,11 +11652,12 @@ static void __init net_dev_struct_check(void)
CACHELINE_ASSERT_GROUP_SIZE(struct net_device, net_device_read_tx, 160);
/* TXRX read-mostly hotpath */
+ CACHELINE_ASSERT_GROUP_MEMBER(struct net_device, net_device_read_txrx, lstats);
CACHELINE_ASSERT_GROUP_MEMBER(struct net_device, net_device_read_txrx, flags);
CACHELINE_ASSERT_GROUP_MEMBER(struct net_device, net_device_read_txrx, hard_header_len);
CACHELINE_ASSERT_GROUP_MEMBER(struct net_device, net_device_read_txrx, features);
CACHELINE_ASSERT_GROUP_MEMBER(struct net_device, net_device_read_txrx, ip6_ptr);
- CACHELINE_ASSERT_GROUP_SIZE(struct net_device, net_device_read_txrx, 30);
+ CACHELINE_ASSERT_GROUP_SIZE(struct net_device, net_device_read_txrx, 38);
/* RX read-mostly hotpath */
CACHELINE_ASSERT_GROUP_MEMBER(struct net_device, net_device_read_rx, ptype_specific);
--
2.43.0.594.gd9cf4e227d-goog
^ permalink raw reply related [flat|nested] 8+ messages in thread* Re: [PATCH net 3/3] net-device: move lstats in net_device_read_txrx
2024-02-08 14:43 ` [PATCH net 3/3] net-device: move lstats in net_device_read_txrx Eric Dumazet
@ 2024-02-09 17:15 ` Simon Horman
0 siblings, 0 replies; 8+ messages in thread
From: Simon Horman @ 2024-02-09 17:15 UTC (permalink / raw)
To: Eric Dumazet
Cc: David S . Miller, Jakub Kicinski, Paolo Abeni, netdev,
Naman Gulati, Coco Li, Wei Wang, eric.dumazet
On Thu, Feb 08, 2024 at 02:43:23PM +0000, Eric Dumazet wrote:
> dev->lstats is notably used from loopback ndo_start_xmit()
> and other virtual drivers.
>
> Per cpu stats updates are dirtying per-cpu data,
> but the pointer itself is read-only.
>
> Fixes: 43a71cd66b9c ("net-device: reorganize net_device fast path variables")
> Signed-off-by: Eric Dumazet <edumazet@google.com>
> Cc: Coco Li <lixiaoyan@google.com>
> Cc: Simon Horman <horms@kernel.org>
Reviewed-by: Simon Horman <horms@kernel.org>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH net 0/3] net: more three misplaced fields
2024-02-08 14:43 [PATCH net 0/3] net: more three misplaced fields Eric Dumazet
` (2 preceding siblings ...)
2024-02-08 14:43 ` [PATCH net 3/3] net-device: move lstats in net_device_read_txrx Eric Dumazet
@ 2024-02-12 10:00 ` patchwork-bot+netdevbpf
3 siblings, 0 replies; 8+ messages in thread
From: patchwork-bot+netdevbpf @ 2024-02-12 10:00 UTC (permalink / raw)
To: Eric Dumazet
Cc: davem, kuba, pabeni, netdev, namangulati, lixiaoyan, weiwan,
eric.dumazet
Hello:
This series was applied to netdev/net.git (main)
by David S. Miller <davem@davemloft.net>:
On Thu, 8 Feb 2024 14:43:20 +0000 you wrote:
> We recently reorganized some structures for better data locality
> in networking fast paths.
>
> This series moves three fields that were not correctly classified.
>
> There probably more to come.
>
> [...]
Here is the summary with links:
- [net,1/3] tcp: move tp->scaling_ratio to tcp_sock_read_txrx group
https://git.kernel.org/netdev/net/c/119ff04864a2
- [net,2/3] tcp: move tp->tcp_usec_ts to tcp_sock_read_txrx group
https://git.kernel.org/netdev/net/c/666a877deab2
- [net,3/3] net-device: move lstats in net_device_read_txrx
https://git.kernel.org/netdev/net/c/c353c7b7ffb7
You are awesome, thank you!
--
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html
^ permalink raw reply [flat|nested] 8+ messages in thread