* [PATCH net-next v2 1/4] tools: ynl: move the new line in NlMsg __repr__
2024-03-05 5:33 [PATCH net-next v2 0/4] tools: ynl: add --dbg-small-recv for easier kernel testing Jakub Kicinski
@ 2024-03-05 5:33 ` Jakub Kicinski
2024-03-05 5:33 ` [PATCH net-next v2 2/4] tools: ynl: allow setting recv() size Jakub Kicinski
` (3 subsequent siblings)
4 siblings, 0 replies; 8+ messages in thread
From: Jakub Kicinski @ 2024-03-05 5:33 UTC (permalink / raw)
To: davem; +Cc: netdev, edumazet, pabeni, jiri, donald.hunter, Jakub Kicinski
We add the new line even if message has no error or extack,
which leads to print(nl_msg) ending with two new lines.
Reviewed-by: Donald Hunter <donald.hunter@gmail.com>
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
---
tools/net/ynl/lib/ynl.py | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/tools/net/ynl/lib/ynl.py b/tools/net/ynl/lib/ynl.py
index ac55aa5a3083..92ade9105f31 100644
--- a/tools/net/ynl/lib/ynl.py
+++ b/tools/net/ynl/lib/ynl.py
@@ -213,11 +213,11 @@ from .nlspec import SpecFamily
return self.nl_type
def __repr__(self):
- msg = f"nl_len = {self.nl_len} ({len(self.raw)}) nl_flags = 0x{self.nl_flags:x} nl_type = {self.nl_type}\n"
+ msg = f"nl_len = {self.nl_len} ({len(self.raw)}) nl_flags = 0x{self.nl_flags:x} nl_type = {self.nl_type}"
if self.error:
- msg += '\terror: ' + str(self.error)
+ msg += '\n\terror: ' + str(self.error)
if self.extack:
- msg += '\textack: ' + repr(self.extack)
+ msg += '\n\textack: ' + repr(self.extack)
return msg
--
2.44.0
^ permalink raw reply related [flat|nested] 8+ messages in thread* [PATCH net-next v2 2/4] tools: ynl: allow setting recv() size
2024-03-05 5:33 [PATCH net-next v2 0/4] tools: ynl: add --dbg-small-recv for easier kernel testing Jakub Kicinski
2024-03-05 5:33 ` [PATCH net-next v2 1/4] tools: ynl: move the new line in NlMsg __repr__ Jakub Kicinski
@ 2024-03-05 5:33 ` Jakub Kicinski
2024-03-05 11:04 ` Donald Hunter
2024-03-05 5:33 ` [PATCH net-next v2 3/4] tools: ynl: support debug printing messages Jakub Kicinski
` (2 subsequent siblings)
4 siblings, 1 reply; 8+ messages in thread
From: Jakub Kicinski @ 2024-03-05 5:33 UTC (permalink / raw)
To: davem; +Cc: netdev, edumazet, pabeni, jiri, donald.hunter, Jakub Kicinski
Make the size of the buffer we use for recv() configurable.
The details of the buffer sizing in netlink are somewhat
arcane, we could spend a lot of time polishing this API.
Let's just leave some hopefully helpful comments for now.
This is a for-developers-only feature, anyway.
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
---
v2:
- move the handling somewhere a bit more sensible
- use 0 as default, and if to set it
---
tools/net/ynl/lib/ynl.py | 21 ++++++++++++++++++---
1 file changed, 18 insertions(+), 3 deletions(-)
diff --git a/tools/net/ynl/lib/ynl.py b/tools/net/ynl/lib/ynl.py
index 92ade9105f31..c3ff5be33e4e 100644
--- a/tools/net/ynl/lib/ynl.py
+++ b/tools/net/ynl/lib/ynl.py
@@ -84,6 +84,10 @@ from .nlspec import SpecFamily
return f"Netlink error: {os.strerror(-self.nl_msg.error)}\n{self.nl_msg}"
+class ConfigError(Exception):
+ pass
+
+
class NlAttr:
ScalarFormat = namedtuple('ScalarFormat', ['native', 'big', 'little'])
type_formats = {
@@ -400,7 +404,8 @@ genl_family_name_to_id = None
class YnlFamily(SpecFamily):
- def __init__(self, def_path, schema=None, process_unknown=False):
+ def __init__(self, def_path, schema=None, process_unknown=False,
+ recv_size=0):
super().__init__(def_path, schema)
self.include_raw = False
@@ -415,6 +420,16 @@ genl_family_name_to_id = None
except KeyError:
raise Exception(f"Family '{self.yaml['name']}' not supported by the kernel")
+ # Note that netlink will use conservative (min) message size for
+ # the first dump recv() on the socket, our setting will only matter
+ # from the second recv() on.
+ self._recv_size = recv_size if recv_size else 131072
+ # Netlink will always allocate at least PAGE_SIZE - sizeof(skb_shinfo)
+ # for a message, so smaller receive sizes will lead to truncation.
+ # Note that the min size for other families may be larger than 4k!
+ if self._recv_size < 4000:
+ raise ConfigError()
+
self.sock = socket.socket(socket.AF_NETLINK, socket.SOCK_RAW, self.nlproto.proto_num)
self.sock.setsockopt(Netlink.SOL_NETLINK, Netlink.NETLINK_CAP_ACK, 1)
self.sock.setsockopt(Netlink.SOL_NETLINK, Netlink.NETLINK_EXT_ACK, 1)
@@ -799,7 +814,7 @@ genl_family_name_to_id = None
def check_ntf(self):
while True:
try:
- reply = self.sock.recv(128 * 1024, socket.MSG_DONTWAIT)
+ reply = self.sock.recv(self._recv_size, socket.MSG_DONTWAIT)
except BlockingIOError:
return
@@ -854,7 +869,7 @@ genl_family_name_to_id = None
done = False
rsp = []
while not done:
- reply = self.sock.recv(128 * 1024)
+ reply = self.sock.recv(self._recv_size)
nms = NlMsgs(reply, attr_space=op.attr_set)
for nl_msg in nms:
if nl_msg.extack:
--
2.44.0
^ permalink raw reply related [flat|nested] 8+ messages in thread* [PATCH net-next v2 3/4] tools: ynl: support debug printing messages
2024-03-05 5:33 [PATCH net-next v2 0/4] tools: ynl: add --dbg-small-recv for easier kernel testing Jakub Kicinski
2024-03-05 5:33 ` [PATCH net-next v2 1/4] tools: ynl: move the new line in NlMsg __repr__ Jakub Kicinski
2024-03-05 5:33 ` [PATCH net-next v2 2/4] tools: ynl: allow setting recv() size Jakub Kicinski
@ 2024-03-05 5:33 ` Jakub Kicinski
2024-03-05 5:33 ` [PATCH net-next v2 4/4] tools: ynl: add --dbg-small-recv for easier kernel testing Jakub Kicinski
2024-03-06 12:10 ` [PATCH net-next v2 0/4] " patchwork-bot+netdevbpf
4 siblings, 0 replies; 8+ messages in thread
From: Jakub Kicinski @ 2024-03-05 5:33 UTC (permalink / raw)
To: davem; +Cc: netdev, edumazet, pabeni, jiri, donald.hunter, Jakub Kicinski
For manual debug, allow printing the netlink level messages
to stderr.
Reviewed-by: Donald Hunter <donald.hunter@gmail.com>
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
---
tools/net/ynl/lib/ynl.py | 15 +++++++++++++++
1 file changed, 15 insertions(+)
diff --git a/tools/net/ynl/lib/ynl.py b/tools/net/ynl/lib/ynl.py
index c3ff5be33e4e..239e22b7a85f 100644
--- a/tools/net/ynl/lib/ynl.py
+++ b/tools/net/ynl/lib/ynl.py
@@ -7,6 +7,7 @@ import random
import socket
import struct
from struct import Struct
+import sys
import yaml
import ipaddress
import uuid
@@ -420,6 +421,7 @@ genl_family_name_to_id = None
except KeyError:
raise Exception(f"Family '{self.yaml['name']}' not supported by the kernel")
+ self._recv_dbg = False
# Note that netlink will use conservative (min) message size for
# the first dump recv() on the socket, our setting will only matter
# from the second recv() on.
@@ -453,6 +455,17 @@ genl_family_name_to_id = None
self.sock.setsockopt(Netlink.SOL_NETLINK, Netlink.NETLINK_ADD_MEMBERSHIP,
mcast_id)
+ def set_recv_dbg(self, enabled):
+ self._recv_dbg = enabled
+
+ def _recv_dbg_print(self, reply, nl_msgs):
+ if not self._recv_dbg:
+ return
+ print("Recv: read", len(reply), "bytes,",
+ len(nl_msgs.msgs), "messages", file=sys.stderr)
+ for nl_msg in nl_msgs:
+ print(" ", nl_msg, file=sys.stderr)
+
def _encode_enum(self, attr_spec, value):
enum = self.consts[attr_spec['enum']]
if enum.type == 'flags' or attr_spec.get('enum-as-flags', False):
@@ -819,6 +832,7 @@ genl_family_name_to_id = None
return
nms = NlMsgs(reply)
+ self._recv_dbg_print(reply, nms)
for nl_msg in nms:
if nl_msg.error:
print("Netlink error in ntf!?", os.strerror(-nl_msg.error))
@@ -871,6 +885,7 @@ genl_family_name_to_id = None
while not done:
reply = self.sock.recv(self._recv_size)
nms = NlMsgs(reply, attr_space=op.attr_set)
+ self._recv_dbg_print(reply, nms)
for nl_msg in nms:
if nl_msg.extack:
self._decode_extack(msg, op, nl_msg.extack)
--
2.44.0
^ permalink raw reply related [flat|nested] 8+ messages in thread* [PATCH net-next v2 4/4] tools: ynl: add --dbg-small-recv for easier kernel testing
2024-03-05 5:33 [PATCH net-next v2 0/4] tools: ynl: add --dbg-small-recv for easier kernel testing Jakub Kicinski
` (2 preceding siblings ...)
2024-03-05 5:33 ` [PATCH net-next v2 3/4] tools: ynl: support debug printing messages Jakub Kicinski
@ 2024-03-05 5:33 ` Jakub Kicinski
2024-03-05 11:05 ` Donald Hunter
2024-03-06 12:10 ` [PATCH net-next v2 0/4] " patchwork-bot+netdevbpf
4 siblings, 1 reply; 8+ messages in thread
From: Jakub Kicinski @ 2024-03-05 5:33 UTC (permalink / raw)
To: davem; +Cc: netdev, edumazet, pabeni, jiri, donald.hunter, Jakub Kicinski
Most "production" netlink clients use large buffers to
make dump efficient, which means that handling of dump
continuation in the kernel is not very well tested.
Add an option for debugging / testing handling of dumps.
It enables printing of extra netlink-level debug and
lowers the recv() buffer size in one go. When used
without any argument (--dbg-small-recv) it picks
a very small default (4000), explicit size can be set,
too (--dbg-small-recv 5000).
Example:
$ ./cli.py [...] --dbg-small-recv
Recv: read 3712 bytes, 29 messages
nl_len = 128 (112) nl_flags = 0x0 nl_type = 19
[...]
nl_len = 128 (112) nl_flags = 0x0 nl_type = 19
Recv: read 3968 bytes, 31 messages
nl_len = 128 (112) nl_flags = 0x0 nl_type = 19
[...]
nl_len = 128 (112) nl_flags = 0x0 nl_type = 19
Recv: read 532 bytes, 5 messages
nl_len = 128 (112) nl_flags = 0x0 nl_type = 19
[...]
nl_len = 128 (112) nl_flags = 0x0 nl_type = 19
nl_len = 20 (4) nl_flags = 0x2 nl_type = 3
(the [...] are edits to shorten the commit message).
Note that the first message of the dump is sized conservatively
by the kernel.
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
---
tools/net/ynl/cli.py | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/tools/net/ynl/cli.py b/tools/net/ynl/cli.py
index 0f8239979670..e8a65fbc3698 100755
--- a/tools/net/ynl/cli.py
+++ b/tools/net/ynl/cli.py
@@ -38,6 +38,8 @@ from lib import YnlFamily, Netlink
const=Netlink.NLM_F_APPEND)
parser.add_argument('--process-unknown', action=argparse.BooleanOptionalAction)
parser.add_argument('--output-json', action='store_true')
+ parser.add_argument('--dbg-small-recv', default=0, const=4000,
+ action='store', nargs='?', type=int)
args = parser.parse_args()
def output(msg):
@@ -53,7 +55,10 @@ from lib import YnlFamily, Netlink
if args.json_text:
attrs = json.loads(args.json_text)
- ynl = YnlFamily(args.spec, args.schema, args.process_unknown)
+ ynl = YnlFamily(args.spec, args.schema, args.process_unknown,
+ recv_size=args.dbg_small_recv)
+ if args.dbg_small_recv:
+ ynl.set_recv_dbg(True)
if args.ntf:
ynl.ntf_subscribe(args.ntf)
--
2.44.0
^ permalink raw reply related [flat|nested] 8+ messages in thread* Re: [PATCH net-next v2 4/4] tools: ynl: add --dbg-small-recv for easier kernel testing
2024-03-05 5:33 ` [PATCH net-next v2 4/4] tools: ynl: add --dbg-small-recv for easier kernel testing Jakub Kicinski
@ 2024-03-05 11:05 ` Donald Hunter
0 siblings, 0 replies; 8+ messages in thread
From: Donald Hunter @ 2024-03-05 11:05 UTC (permalink / raw)
To: Jakub Kicinski; +Cc: davem, netdev, edumazet, pabeni, jiri
Jakub Kicinski <kuba@kernel.org> writes:
> Most "production" netlink clients use large buffers to
> make dump efficient, which means that handling of dump
> continuation in the kernel is not very well tested.
>
> Add an option for debugging / testing handling of dumps.
> It enables printing of extra netlink-level debug and
> lowers the recv() buffer size in one go. When used
> without any argument (--dbg-small-recv) it picks
> a very small default (4000), explicit size can be set,
> too (--dbg-small-recv 5000).
>
> Example:
>
> $ ./cli.py [...] --dbg-small-recv
> Recv: read 3712 bytes, 29 messages
> nl_len = 128 (112) nl_flags = 0x0 nl_type = 19
> [...]
> nl_len = 128 (112) nl_flags = 0x0 nl_type = 19
> Recv: read 3968 bytes, 31 messages
> nl_len = 128 (112) nl_flags = 0x0 nl_type = 19
> [...]
> nl_len = 128 (112) nl_flags = 0x0 nl_type = 19
> Recv: read 532 bytes, 5 messages
> nl_len = 128 (112) nl_flags = 0x0 nl_type = 19
> [...]
> nl_len = 128 (112) nl_flags = 0x0 nl_type = 19
> nl_len = 20 (4) nl_flags = 0x2 nl_type = 3
>
> (the [...] are edits to shorten the commit message).
>
> Note that the first message of the dump is sized conservatively
> by the kernel.
>
> Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Reviewed-by: Donald Hunter <donald.hunter@gmail.com>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH net-next v2 0/4] tools: ynl: add --dbg-small-recv for easier kernel testing
2024-03-05 5:33 [PATCH net-next v2 0/4] tools: ynl: add --dbg-small-recv for easier kernel testing Jakub Kicinski
` (3 preceding siblings ...)
2024-03-05 5:33 ` [PATCH net-next v2 4/4] tools: ynl: add --dbg-small-recv for easier kernel testing Jakub Kicinski
@ 2024-03-06 12:10 ` patchwork-bot+netdevbpf
4 siblings, 0 replies; 8+ messages in thread
From: patchwork-bot+netdevbpf @ 2024-03-06 12:10 UTC (permalink / raw)
To: Jakub Kicinski; +Cc: davem, netdev, edumazet, pabeni, jiri, donald.hunter
Hello:
This series was applied to netdev/net-next.git (main)
by David S. Miller <davem@davemloft.net>:
On Mon, 4 Mar 2024 21:33:06 -0800 you wrote:
> When testing netlink dumps I usually hack some user space up
> to constrain its user space buffer size (iproute2, ethtool or ynl).
> Netlink will try to fill the messages up, so since these apps use
> large buffers by default, the dumps are rarely fragmented.
>
> I was hoping to figure out a way to create a selftest for dump
> testing, but so far I have no idea how to do that in a useful
> and generic way.
>
> [...]
Here is the summary with links:
- [net-next,v2,1/4] tools: ynl: move the new line in NlMsg __repr__
https://git.kernel.org/netdev/net-next/c/7df7231d6a6b
- [net-next,v2,2/4] tools: ynl: allow setting recv() size
https://git.kernel.org/netdev/net-next/c/7c93a88785da
- [net-next,v2,3/4] tools: ynl: support debug printing messages
https://git.kernel.org/netdev/net-next/c/a6a41521f95e
- [net-next,v2,4/4] tools: ynl: add --dbg-small-recv for easier kernel testing
https://git.kernel.org/netdev/net-next/c/c0111878d45e
You are awesome, thank you!
--
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html
^ permalink raw reply [flat|nested] 8+ messages in thread