netdev.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [PATCH 0/1] Issue description and debug
@ 2023-08-25  7:55 Heng Guo
  2023-08-25  7:55 ` [PATCH 1/1] net: ipv4, ipv6: fix IPSTATS_MIB_OUTOCTETS increment duplicated Heng Guo
  2023-08-30  8:50 ` [PATCH 0/1] Issue description and debug patchwork-bot+netdevbpf
  0 siblings, 2 replies; 5+ messages in thread
From: Heng Guo @ 2023-08-25  7:55 UTC (permalink / raw)
  To: davem, sahern, edumazet, kuba, pabeni
  Cc: netdev, Richard.Danter, filip.pudak, heng.guo

Hi maintainers,

The IPSTATS_MIB_OUTOCTETS increment is duplicated in SNMP test.

Reproduce environment:
network with 3 VM linuxs is connected as below:
VM1<---->VM2(latest kernel 6.5.0-rc7)<---->VM3
VM1: eth0 ip: 192.168.122.207
VM2: eth0 ip: 192.168.122.208, eth1 ip: 192.168.123.104
VM3: eth0 ip: 192.168.123.240

Reproduce:
VM1 send 1400 bytes UDP data to VM3 using tools scapy.
scapy command:
send(IP(dst="192.168.123.240",flags=0)/UDP()/str('0'*1400),count=1,
inter=1.000000)

Result:
----------------------------------------------------------------------
root@qemux86-64:~# cat /proc/net/netstat
TcpExt: SyncookiesSent SyncookiesRecv SyncookiesFailed ......
TcpExt: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  ......
IpExt: InNoRoutes InTruncatedPkts InMcastPkts OutMcastPkts
	InBcastPkts OutBcastPkts InOctets OutOctets    ......
IpExt: 0 0 0 0 0 0 796 140 0 0 0 0 0 4 0 0 0 0
root@qemux86-64:~# 
root@qemux86-64:~# cat /proc/net/netstat
TcpExt: SyncookiesSent SyncookiesRecv SyncookiesFailed ......
TcpExt: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  ......
IpExt: InNoRoutes InTruncatedPkts InMcastPkts OutMcastPkts
	InBcastPkts OutBcastPkts InOctets OutOctets    ......
IpExt: 0 0 0 0 0 0 2224 2996 0 0 0 0 0 5 0 0 0 0
----------------------------------------------------------------------
OutOctets increment is duplicated (2996-140=2856)


Issue description and debug:
Add dump_stack() in ip_output(), get below test logs with core stack.
----------------------------------------------------------------------
root@qemux86-64:~# dmesg
[    0.000000] Linux version 6.5.0-rc7-yocto-standard (oe-user@oe-host)
(x86_64-poky-linux-gcc (GCC) 13.2.0, GNU ld (GNU Binutils)
 2.41.0.20230731) #1 SMP PREEMPT_DY3
[    0.000000] Command line: root=/dev/vda rw console=ttyS0 mem=256M
ip=dhcp
[    0.000000] BIOS-provided physical RAM map:
......
[   64.225063] ip_forward_finish: IPSTATS_MIB_OUTOCTETS len:1428
[   64.228672] ip_output: increase IPSTATS_MIB_OUT len:1428
[   64.228695] CPU: 0 PID: 0 Comm: swapper/0 Not tainted
6.5.0-rc7-yocto-standard #1
[   64.228709] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS 1.13.0-1ubuntu1.1 04/01/2014
[   64.228715] Call Trace:
[   64.228721]  <IRQ>
[   64.228728]  dump_stack_lvl+0x3b/0x50
[   64.228744]  dump_stack+0x14/0x20
[   64.228756]  ip_output+0x129/0x140
[   64.228770]  ? dump_stack_lvl+0x47/0x50
[   64.228785]  ip_forward_finish+0xc8/0xe0
[   64.228801]  ip_forward+0x540/0x5a0
[   64.228819]  ip_sublist_rcv_finish+0x72/0x80
[   64.228834]  ip_sublist_rcv+0x160/0x1a0
[   64.228854]  ip_list_rcv+0xff/0x130
[   64.228871]  __netif_receive_skb_list_core+0x21b/0x240
[   64.228891]  netif_receive_skb_list_internal+0x1b9/0x2d0
[   64.228935]  ? detach_buf_split+0x76/0x150
[   64.228966]  napi_complete_done+0x78/0x190
[   64.228979]  virtnet_poll+0x48f/0x5e0
[   64.229001]  __napi_poll+0x31/0x1d0
[   64.229014]  net_rx_action+0x295/0x300
[   64.229030]  __do_softirq+0xff/0x315
[   64.229047]  irq_exit_rcu+0x82/0xd0
[   64.229059]  common_interrupt+0xb5/0xd0
[   64.229072]  </IRQ>
[   64.229077]  <TASK>
[   64.229083]  asm_common_interrupt+0x2b/0x40
[   64.229095] RIP: 0010:default_idle+0x17/0x20
[   64.229107] Code: ff ff ff 90 90 90 90 90 90 90 90 90 90 90 90 90 90
90 90 f3 0f 1e fa 8b 05 0e f9 e7 00 85 c0 7e 07 0f 00 2d 5b 7c 2c 00 fb
f4 <fa> c3 cc cc cc c0
[   64.229118] RSP: 0018:ffffffff8d203e38 EFLAGS: 00000246
[   64.229130] RAX: 0000000000000000 RBX: 0000000000000000 RCX:
0000000000000000
[   64.229135] RDX: 0000000000000001 RSI: ffffffff8cf9b570 RDI:
0000000000006554
[   64.229141] RBP: ffffffff8d203e40 R08: ffff9ea94f81f420 R09:
0000000000000001
[   64.229147] R10: ffff9ea941226e80 R11: 00000000000000b2 R12:
ffffffff8d20a940
[   64.229153] R13: 0000000000000000 R14: 0000000000000000 R15:
ffffffff8d20a940
[   64.229169]  ? arch_cpu_idle+0xd/0x20
[   64.229182]  default_idle_call+0x36/0xf0
[   64.229194]  do_idle+0x1f1/0x230
[   64.229208]  cpu_startup_entry+0x21/0x30
[   64.229220]  rest_init+0xc5/0xd0
[   64.229233]  arch_call_rest_init+0x12/0x50
[   64.229246]  start_kernel+0x426/0x680
[   64.229262]  x86_64_start_reservations+0x1c/0x30
[   64.229276]  x86_64_start_kernel+0xce/0xe0
[   64.229290]  secondary_startup_64_no_verify+0x179/0x17b
[   64.229338]  </TASK>
[   64.232465] net_dev_queue: { len:1428, name:eth1,
network_header_type: IPV4, protocol: icmp,
saddr:[192.168.122.1], daddr:[192.168.123.240] }
--------------------------------------------------------------------
Refer to source codes, IPSTATS_MIB_OUTOCTETS is counted in both
ip_forward_finish() and ip_output().
commit edf391ff1723 ("snmp: add missing counters for RFC 4293") had
already added OutOctets for RFC 4293. In commit 2d8dbb04c63e ("snmp: fix
OutOctets counter to include forwarded datagrams"), OutOctets was
counted again, but not removed from ip_output().
And according to RFC 4293 "3.2.3. IP Statistics Tables",
ipIfStatsOutTransmits is not same to ipIfStatsOutForwDatagrams. So
"IPSTATS_MIB_OUTOCTETS must be incremented when incrementing
IPSTATS_MIB_OUTFORWDATAGRAMS" is not accurate. And IPSTATS_MIB_OUTOCTETS
should aslo be counted after fragment by RFC 4293.

Patch:
So do patch to revert this comment and to move IPSTATS_MIB_OUTOCTETS to
ip_finish_output2 for ipv4. Test result is:
----------------------------------------------------------------------
root@qemux86-64:~# cat /proc/net/netstat
TcpExt: SyncookiesSent SyncookiesRecv SyncookiesFailed          ......
TcpExt: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ......
IpExt: InNoRoutes InTruncatedPkts InMcastPkts OutMcastPkts InBcastPkts
OutBcastPkts InOctets OutOctets					......
IpExt: 0 0 0 0 1 0 1500 516 0 0 328 0 0 7 0 0 0 0
root@qemux86-64:~# 
root@qemux86-64:~# 
root@qemux86-64:~# cat /proc/net/netstat
TcpExt: SyncookiesSent SyncookiesRecv SyncookiesFailed          ......
TcpExt: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ......
IpExt: InNoRoutes InTruncatedPkts InMcastPkts OutMcastPkts InBcastPkts
OutBcastPkts InOctets OutOctets                                 ......
IpExt: 0 0 0 0 1 0 2928 1944 0 0 328 0 0 8 0 0 0 0
root@qemux86-64:~# dmesg
----------------------------------------------------------------------
OutOctets increment is only 1428(1944-516).


Thanks,
Heng


Heng Guo (1):
  net: ipv4, ipv6: fix IPSTATS_MIB_OUTOCTETS increment duplicated

 net/ipv4/ip_forward.c | 1 -
 net/ipv4/ip_output.c  | 7 +++----
 net/ipv4/ipmr.c       | 1 -
 net/ipv6/ip6_output.c | 1 -
 net/ipv6/ip6mr.c      | 2 --
 5 files changed, 3 insertions(+), 9 deletions(-)

-- 
2.35.5


^ permalink raw reply	[flat|nested] 5+ messages in thread
* [PATCH 0/1] Issue description and debug
@ 2023-09-14  5:16 Heng Guo
  0 siblings, 0 replies; 5+ messages in thread
From: Heng Guo @ 2023-09-14  5:16 UTC (permalink / raw)
  To: davem, sahern, edumazet, kuba, pabeni; +Cc: netdev, filip.pudak, heng.guo

Hi maintainers,

The IPSTATS_MIB_OUTFORWDATAGRAMS is counted after fragment check.

Reproduce environment:
network with 3 VM linuxs is connected as below:
VM1<---->VM2(latest kernel 6.5.0-rc7)<---->VM3
VM1: eth0 ip: 192.168.122.207 MTU 1800
VM2: eth0 ip: 192.168.122.208, eth1 ip: 192.168.123.224 MTU 1500
VM3: eth0 ip: 192.168.123.240 MTU 1800

Reproduce:
VM1 send 1600 bytes UDP data to VM3 using tools scapy with flags='DF'.
scapy command:
send(IP(dst="192.168.123.240",flags='DF')/UDP()/str('0'*1600),count=1,
inter=1.000000)

Result:
Before IP data is sent.
----------------------------------------------------------------------
root@qemux86-64:~# cat /proc/net/snmp
Ip: Forwarding DefaultTTL InReceives InHdrErrors InAddrErrors
ForwDatagrams InUnknownProtos InDiscards InDelivers OutRequests
 OutDiscards OutNoRoutes ReasmTimeout ReasmReqdss
Ip: 1 64 6 0 2 2 0 0 2 4 0 0 0 0 0 0 0 0 0
......
root@qemux86-64:~#
----------------------------------------------------------------------
After IP data is sent.
----------------------------------------------------------------------
root@qemux86-64:~# cat /proc/net/snmp
Ip: Forwarding DefaultTTL InReceives InHdrErrors InAddrErrors
 ForwDatagrams InUnknownProtos InDiscards InDelivers OutRequests
 OutDiscards OutNoRoutes ReasmTimeout ReasmReqdss
Ip: 1 64 7 0 2 2 0 0 2 5 0 0 0 0 0 0 0 1 0
......
root@qemux86-64:~#
----------------------------------------------------------------------
ForwDatagrams is always keeping 2 without increment.

Issue description and patch:
ip_exceeds_mtu() in ip_forward() drops this IP datagram because skb len
(1600 sending by scapy) is over MTU(1500 in VM2) and "DF" is set.
According to RFC 4293 "3.2.3. IP Statistics Tables",
  +-------+------>------+----->-----+----->-----+
  | InForwDatagrams (6) | OutForwDatagrams (6)  |
  |                     V                       +->-+ OutFragReqds
  |                 InNoRoutes                  |   | (packets)
  / (local packet (3)                           |   |
  |  IF is that of the address                  |   +--> OutFragFails
  |  and may not be the receiving IF)           |   |    (packets)
the IPSTATS_MIB_OUTFORWDATAGRAMS should be counted before fragment
check.
The existing implementation, instead, would incease the counter after
fragment check: ip_exceeds_mtu() in ipv4 and ip6_pkt_too_big() in ipv6.
So do patch to move IPSTATS_MIB_OUTFORWDATAGRAMS counter to ip_forward()
for ipv4 and ip6_forward() for ipv6.

Test result with patch:
Before IP data is sent.
----------------------------------------------------------------------
root@qemux86-64:~# cat /proc/net/snmp
Ip: Forwarding DefaultTTL InReceives InHdrErrors InAddrErrors
 ForwDatagrams InUnknownProtos InDiscards InDelivers OutRequests
 OutDiscards OutNoRoutes ReasmTimeout ReasmReqdss
Ip: 1 64 6 0 2 2 0 0 2 4 0 0 0 0 0 0 0 0 0
......
root@qemux86-64:~#
----------------------------------------------------------------------
After IP data is sent.
----------------------------------------------------------------------
root@qemux86-64:~# cat /proc/net/snmp
Ip: Forwarding DefaultTTL InReceives InHdrErrors InAddrErrors
 ForwDatagrams InUnknownProtos InDiscards InDelivers OutRequests
 OutDiscards OutNoRoutes ReasmTimeout ReasmReqdss
Ip: 1 64 7 0 2 3 0 0 2 5 0 0 0 0 0 0 0 1 0
......
root@qemux86-64:~#
----------------------------------------------------------------------
ForwDatagrams is updated from 2 to 3.

Thanks,
Heng


Heng Guo (1):
  net: ipv4,ipv6: fix IPSTATS_MIB_OUTFORWDATAGRAMS increment after
    fragment check

 net/ipv4/ip_forward.c | 4 ++--
 net/ipv6/ip6_output.c | 4 ++--
 2 files changed, 4 insertions(+), 4 deletions(-)

-- 
2.25.1


^ permalink raw reply	[flat|nested] 5+ messages in thread
* [PATCH 0/1] Issue description and debug
@ 2023-10-08  0:59 Heng Guo
  0 siblings, 0 replies; 5+ messages in thread
From: Heng Guo @ 2023-10-08  0:59 UTC (permalink / raw)
  To: davem, dsahern, edumazet, kuba, pabeni; +Cc: netdev, filip.pudak, heng.guo

Hi maintainers,

The IPSTATS_MIB_OUTFORWDATAGRAMS is counted after fragment check.

Reproduce environment:
network with 3 VM linuxs is connected as below:
VM1<---->VM2(latest kernel 6.5.0-rc7)<---->VM3
VM1: eth0 ip: 192.168.122.207 MTU 1800
VM2: eth0 ip: 192.168.122.208, eth1 ip: 192.168.123.224 MTU 1500
VM3: eth0 ip: 192.168.123.240 MTU 1800

Reproduce:
VM1 send 1600 bytes UDP data to VM3 using tools scapy with flags='DF'.
scapy command:
send(IP(dst="192.168.123.240",flags='DF')/UDP()/str('0'*1600),count=1,
inter=1.000000)

Result:
Before IP data is sent.
----------------------------------------------------------------------
root@qemux86-64:~# cat /proc/net/snmp
Ip: Forwarding DefaultTTL InReceives InHdrErrors InAddrErrors
ForwDatagrams InUnknownProtos InDiscards InDelivers OutRequests
OutDiscards OutNoRoutes ReasmTimeout ReasmReqdss
Ip: 1 64 6 0 2 2 0 0 2 4 0 0 0 0 0 0 0 0 0
......
root@qemux86-64:~#
----------------------------------------------------------------------
After IP data is sent.
----------------------------------------------------------------------
root@qemux86-64:~# cat /proc/net/snmp
Ip: Forwarding DefaultTTL InReceives InHdrErrors InAddrErrors
ForwDatagrams InUnknownProtos InDiscards InDelivers OutRequests
OutDiscards OutNoRoutes ReasmTimeout ReasmReqdss
Ip: 1 64 7 0 2 2 0 0 2 5 0 0 0 0 0 0 0 1 0
......
root@qemux86-64:~#
----------------------------------------------------------------------
ForwDatagrams is always keeping 2 without increment.

Issue description and patch:
ip_exceeds_mtu() in ip_forward() drops this IP datagram because skb len
(1600 sending by scapy) is over MTU(1500 in VM2) and "DF" is set.
According to RFC 4293 "3.2.3. IP Statistics Tables",
  +-------+------>------+----->-----+----->-----+
  | InForwDatagrams (6) | OutForwDatagrams (6)  |
  |                     V                       +->-+ OutFragReqds
  |                 InNoRoutes                  |   | (packets)
  / (local packet (3)                           |   |
  |  IF is that of the address                  |   +--> OutFragFails
  |  and may not be the receiving IF)           |   |    (packets)
the IPSTATS_MIB_OUTFORWDATAGRAMS should be counted before fragment
check.
The existing implementation, instead, would incease the counter after
fragment check: ip_exceeds_mtu() in ipv4 and ip6_pkt_too_big() in ipv6.
So do patch to move IPSTATS_MIB_OUTFORWDATAGRAMS counter to ip_forward()
for ipv4 and ip6_forward() for ipv6.

Test result with patch:
Before IP data is sent.
----------------------------------------------------------------------
root@qemux86-64:~# cat /proc/net/snmp
Ip: Forwarding DefaultTTL InReceives InHdrErrors InAddrErrors
 ForwDatagrams InUnknownProtos InDiscards InDelivers OutRequests
 OutDiscards OutNoRoutes ReasmTimeout ReasmReqdss
Ip: 1 64 6 0 2 2 0 0 2 4 0 0 0 0 0 0 0 0 0
......
root@qemux86-64:~#
----------------------------------------------------------------------
After IP data is sent.
----------------------------------------------------------------------
root@qemux86-64:~# cat /proc/net/snmp
Ip: Forwarding DefaultTTL InReceives InHdrErrors InAddrErrors
 ForwDatagrams InUnknownProtos InDiscards InDelivers OutRequests
 OutDiscards OutNoRoutes ReasmTimeout ReasmReqdss
Ip: 1 64 7 0 2 3 0 0 2 5 0 0 0 0 0 0 0 1 0
......
root@qemux86-64:~#
----------------------------------------------------------------------
ForwDatagrams is updated from 2 to 3.

Thanks,
Heng

This is a new email thread, last one is:
https://lore.kernel.org/all/20230914051623.2180843-2-heng.guo@windriver.com


Heng Guo (1):
  net-next: fix IPSTATS_MIB_OUTFORWDATAGRAMS increment after fragment
    check

 net/ipv4/ip_forward.c | 4 ++--
 net/ipv6/ip6_output.c | 6 ++----
 2 files changed, 4 insertions(+), 6 deletions(-)

-- 
2.25.1


^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2023-10-08  0:59 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2023-08-25  7:55 [PATCH 0/1] Issue description and debug Heng Guo
2023-08-25  7:55 ` [PATCH 1/1] net: ipv4, ipv6: fix IPSTATS_MIB_OUTOCTETS increment duplicated Heng Guo
2023-08-30  8:50 ` [PATCH 0/1] Issue description and debug patchwork-bot+netdevbpf
  -- strict thread matches above, loose matches on Subject: below --
2023-09-14  5:16 Heng Guo
2023-10-08  0:59 Heng Guo

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).