* Re: WARNING: refcount bug in should_fail
From: Tetsuo Handa @ 2018-04-01 11:11 UTC (permalink / raw)
To: dvyukov
Cc: viro, ebiederm, linux-fsdevel, netdev, linux-mm, syzkaller-bugs,
syzbot+84371b6062cb639d797e
In-Reply-To: <CACT4Y+ZgkV5BF24BYptq5rDYo0oTJq4oHw6_808vLTmMZBmE8A@mail.gmail.com>
Dmitry Vyukov wrote:
> On Sun, Apr 1, 2018 at 12:32 PM, Dmitry Vyukov <dvyukov@google.com> wrote:
> > On Sun, Mar 4, 2018 at 6:57 AM, Tetsuo Handa
> > <penguin-kernel@i-love.sakura.ne.jp> wrote:
> >> Switching from mm to fsdevel, for this report says that put_net(net) in
> >> rpc_kill_sb() made net->count < 0 when mount_ns() failed due to
> >> register_shrinker() failure.
> >>
> >> Relevant commits will be
> >> commit 9ee332d99e4d5a97 ("sget(): handle failures of register_shrinker()") and
> >> commit d91ee87d8d85a080 ("vfs: Pass data, ns, and ns->userns to mount_ns.").
> >>
> >> When sget_userns() in mount_ns() failed, mount_ns() returns an error code to
> >> the caller without calling fill_super(). That is, get_net(sb->s_fs_info) was
> >> not called by rpc_fill_super() (via fill_super callback passed to mount_ns())
> >> but put_net(sb->s_fs_info) is called by rpc_kill_sb() (via fs->kill_sb() from
> >> deactivate_locked_super()).
> >>
> >> ----------
> >> static struct dentry *
> >> rpc_mount(struct file_system_type *fs_type,
> >> int flags, const char *dev_name, void *data)
> >> {
> >> struct net *net = current->nsproxy->net_ns;
> >> return mount_ns(fs_type, flags, data, net, net->user_ns, rpc_fill_super);
> >> }
> >> ----------
> >
> > Messed kernel output, this is definitely not in should_fail.
> >
> > #syz dup: WARNING: refcount bug in sk_alloc
>
> Please don't drop reporter (syzbot) email from CC.
>
> #syz dup: WARNING: refcount bug in sk_alloc
>
Excuse me? This "refcount bug in should_fail" is talking about sget_userns() versus rpc_fill_super().
I think we need to fix either 9ee332d99e4d5a97 or d91ee87d8d85a080.
>
> >> syzbot wrote:
> >>> Hello,
> >>>
> >>> syzbot hit the following crash on bpf-next commit
> >>> 6f1b5a2b58d8470e5a8b25ab29f5fdb4616ffff8 (Tue Feb 27 04:11:23 2018 +0000)
> >>> Merge branch 'bpf-kselftest-improvements'
> >>>
> >>> C reproducer is attached.
> >>> syzkaller reproducer is attached.
> >>> Raw console output is attached.
> >>> compiler: gcc (GCC) 7.1.1 20170620
> >>> .config is attached.
> >>>
> >>> IMPORTANT: if you fix the bug, please add the following tag to the commit:
> >>> Reported-by: syzbot+84371b6062cb639d797e@syzkaller.appspotmail.com
> >>> It will help syzbot understand when the bug is fixed. See footer for
> >>> details.
> >>> If you forward the report, please keep this part and the footer.
> >>>
> >>> ------------[ cut here ]------------
> >>> FAULT_INJECTION: forcing a failure.
> >>> name failslab, interval 1, probability 0, space 0, times 0
> >>> refcount_t: underflow; use-after-free.
> >>> CPU: 1 PID: 4239 Comm: syzkaller149381 Not tainted 4.16.0-rc2+ #20
> >>> WARNING: CPU: 0 PID: 4237 at lib/refcount.c:187
> >>> refcount_sub_and_test+0x167/0x1b0 lib/refcount.c:187
> >>> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
> >>> Google 01/01/2011
> >>> Call Trace:
> >>> Kernel panic - not syncing: panic_on_warn set ...
> >>>
> >>> __dump_stack lib/dump_stack.c:17 [inline]
> >>> dump_stack+0x194/0x24d lib/dump_stack.c:53
> >>> fail_dump lib/fault-inject.c:51 [inline]
> >>> should_fail+0x8c0/0xa40 lib/fault-inject.c:149
> >>> should_failslab+0xec/0x120 mm/failslab.c:32
> >>> slab_pre_alloc_hook mm/slab.h:422 [inline]
> >>> slab_alloc mm/slab.c:3365 [inline]
> >>> __do_kmalloc mm/slab.c:3703 [inline]
> >>> __kmalloc+0x63/0x760 mm/slab.c:3714
> >>> kmalloc include/linux/slab.h:517 [inline]
> >>> kzalloc include/linux/slab.h:701 [inline]
> >>> register_shrinker+0x10e/0x2d0 mm/vmscan.c:268
> >>> sget_userns+0xbbf/0xe40 fs/super.c:520
> >>> mount_ns+0x6d/0x190 fs/super.c:1029
> >>> rpc_mount+0x9e/0xd0 net/sunrpc/rpc_pipe.c:1451
> >>> mount_fs+0x66/0x2d0 fs/super.c:1222
> >>> vfs_kern_mount.part.26+0xc6/0x4a0 fs/namespace.c:1037
> >>> vfs_kern_mount fs/namespace.c:2509 [inline]
> >>> do_new_mount fs/namespace.c:2512 [inline]
> >>> do_mount+0xea4/0x2bb0 fs/namespace.c:2842
> >>> SYSC_mount fs/namespace.c:3058 [inline]
> >>> SyS_mount+0xab/0x120 fs/namespace.c:3035
> >>> do_syscall_64+0x280/0x940 arch/x86/entry/common.c:287
> >>> entry_SYSCALL_64_after_hwframe+0x42/0xb7
> >>> RIP: 0033:0x4460f9
> >>> RSP: 002b:00007fbcd769ad78 EFLAGS: 00000246 ORIG_RAX: 00000000000000a5
> >>> RAX: ffffffffffffffda RBX: 00000000006dcc6c RCX: 00000000004460f9
> >>> RDX: 0000000020000080 RSI: 0000000020000040 RDI: 0000000020000000
> >>> RBP: 00007fbcd769ad80 R08: 00000000200000c0 R09: 0000000000003131
> >>> R10: 0000000000000000 R11: 0000000000000246 R12: 00000000006dcc68
> >>> R13: ffffffffffffffff R14: 0000000000000037 R15: 0030656c69662f2e
> >>> CPU: 0 PID: 4237 Comm: syzkaller149381 Not tainted 4.16.0-rc2+ #20
> >>> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
> >>> Google 01/01/2011
> >>> Call Trace:
> >>> __dump_stack lib/dump_stack.c:17 [inline]
> >>> dump_stack+0x194/0x24d lib/dump_stack.c:53
> >>> panic+0x1e4/0x41c kernel/panic.c:183
> >>> __warn+0x1dc/0x200 kernel/panic.c:547
> >>> report_bug+0x211/0x2d0 lib/bug.c:184
> >>> fixup_bug.part.11+0x37/0x80 arch/x86/kernel/traps.c:178
> >>> fixup_bug arch/x86/kernel/traps.c:247 [inline]
> >>> do_error_trap+0x2d7/0x3e0 arch/x86/kernel/traps.c:296
> >>> do_invalid_op+0x1b/0x20 arch/x86/kernel/traps.c:315
> >>> invalid_op+0x58/0x80 arch/x86/entry/entry_64.S:957
> >>> RIP: 0010:refcount_sub_and_test+0x167/0x1b0 lib/refcount.c:187
> >>> RSP: 0018:ffff8801b164f6d8 EFLAGS: 00010286
> >>> RAX: dffffc0000000008 RBX: 0000000000000000 RCX: ffffffff815ac30e
> >>> RDX: 0000000000000000 RSI: 1ffff100362c9e8b RDI: 1ffff100362c9e60
> >>> RBP: ffff8801b164f768 R08: 0000000000000000 R09: 0000000000000000
> >>> R10: ffff8801b164f610 R11: 0000000000000000 R12: 1ffff100362c9edc
> >>> R13: 00000000ffffffff R14: 0000000000000001 R15: ffff8801ae924044
> >>> refcount_dec_and_test+0x1a/0x20 lib/refcount.c:212
> >>> put_net include/net/net_namespace.h:220 [inline]
> >>> rpc_kill_sb+0x253/0x3c0 net/sunrpc/rpc_pipe.c:1473
> >>> deactivate_locked_super+0x88/0xd0 fs/super.c:312
> >>> sget_userns+0xbda/0xe40 fs/super.c:522
> >>> mount_ns+0x6d/0x190 fs/super.c:1029
> >>> rpc_mount+0x9e/0xd0 net/sunrpc/rpc_pipe.c:1451
> >>> mount_fs+0x66/0x2d0 fs/super.c:1222
> >>> vfs_kern_mount.part.26+0xc6/0x4a0 fs/namespace.c:1037
> >>> vfs_kern_mount fs/namespace.c:2509 [inline]
> >>> do_new_mount fs/namespace.c:2512 [inline]
> >>> do_mount+0xea4/0x2bb0 fs/namespace.c:2842
> >>> SYSC_mount fs/namespace.c:3058 [inline]
> >>> SyS_mount+0xab/0x120 fs/namespace.c:3035
> >>> do_syscall_64+0x280/0x940 arch/x86/entry/common.c:287
> >>> entry_SYSCALL_64_after_hwframe+0x42/0xb7
> >>> RIP: 0033:0x4460f9
> >>> RSP: 002b:00007fbcd76dcd78 EFLAGS: 00000246 ORIG_RAX: 00000000000000a5
> >>> RAX: ffffffffffffffda RBX: 00000000006dcc3c RCX: 00000000004460f9
> >>> RDX: 0000000020000080 RSI: 0000000020000040 RDI: 0000000020000000
> >>> RBP: 00007fbcd76dcd80 R08: 00000000200000c0 R09: 0000000000003131
> >>> R10: 0000000000000000 R11: 0000000000000246 R12: 00000000006dcc38
> >>> R13: ffffffffffffffff R14: 0000000000000028 R15: 0030656c69662f2e
> >>> Dumping ftrace buffer:
> >>> (ftrace buffer empty)
> >>> Kernel Offset: disabled
> >>> Rebooting in 86400 seconds..
> >>>
> >>>
> >>> ---
> >>> This bug is generated by a dumb bot. It may contain errors.
> >>> See https://goo.gl/tpsmEJ for details.
> >>> Direct all questions to syzkaller@googlegroups.com.
> >>>
> >>> syzbot will keep track of this bug report.
> >>> If you forgot to add the Reported-by tag, once the fix for this bug is
> >>> merged
> >>> into any tree, please reply to this email with:
> >>> #syz fix: exact-commit-title
> >>> If you want to test a patch for this bug, please reply with:
> >>> #syz test: git://repo/address.git branch
> >>> and provide the patch inline or as an attachment.
> >>> To mark this as a duplicate of another syzbot report, please reply with:
> >>> #syz dup: exact-subject-of-another-report
> >>> If it's a one-off invalid bug report, please reply with:
> >>> #syz invalid
> >>> Note: if the crash happens again, it will cause creation of a new bug
> >>> report.
> >>> Note: all commands must start from beginning of the line in the email body.
> >>>
> >>
> >> --
> >> You received this message because you are subscribed to the Google Groups "syzkaller-bugs" group.
> >> To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller-bugs+unsubscribe@googlegroups.com.
> >> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller-bugs/201803041457.GBJ69774.OVOSOLFHQMJFFt%40I-love.SAKURA.ne.jp.
> >> For more options, visit https://groups.google.com/d/optout.
>
^ permalink raw reply
* Re: WARNING: refcount bug in should_fail
From: Dmitry Vyukov @ 2018-04-01 11:30 UTC (permalink / raw)
To: Tetsuo Handa
Cc: Al Viro, Eric W. Biederman, linux-fsdevel, netdev, Linux-MM,
syzkaller-bugs, syzbot+84371b6062cb639d797e
In-Reply-To: <201804012011.ABI74044.MJtOFVSHFQFOOL@I-love.SAKURA.ne.jp>
On Sun, Apr 1, 2018 at 1:11 PM, Tetsuo Handa
<penguin-kernel@i-love.sakura.ne.jp> wrote:
> Dmitry Vyukov wrote:
>> On Sun, Apr 1, 2018 at 12:32 PM, Dmitry Vyukov <dvyukov@google.com> wrote:
>> > On Sun, Mar 4, 2018 at 6:57 AM, Tetsuo Handa
>> > <penguin-kernel@i-love.sakura.ne.jp> wrote:
>> >> Switching from mm to fsdevel, for this report says that put_net(net) in
>> >> rpc_kill_sb() made net->count < 0 when mount_ns() failed due to
>> >> register_shrinker() failure.
>> >>
>> >> Relevant commits will be
>> >> commit 9ee332d99e4d5a97 ("sget(): handle failures of register_shrinker()") and
>> >> commit d91ee87d8d85a080 ("vfs: Pass data, ns, and ns->userns to mount_ns.").
>> >>
>> >> When sget_userns() in mount_ns() failed, mount_ns() returns an error code to
>> >> the caller without calling fill_super(). That is, get_net(sb->s_fs_info) was
>> >> not called by rpc_fill_super() (via fill_super callback passed to mount_ns())
>> >> but put_net(sb->s_fs_info) is called by rpc_kill_sb() (via fs->kill_sb() from
>> >> deactivate_locked_super()).
>> >>
>> >> ----------
>> >> static struct dentry *
>> >> rpc_mount(struct file_system_type *fs_type,
>> >> int flags, const char *dev_name, void *data)
>> >> {
>> >> struct net *net = current->nsproxy->net_ns;
>> >> return mount_ns(fs_type, flags, data, net, net->user_ns, rpc_fill_super);
>> >> }
>> >> ----------
>> >
>> > Messed kernel output, this is definitely not in should_fail.
>> >
>> > #syz dup: WARNING: refcount bug in sk_alloc
>>
>> Please don't drop reporter (syzbot) email from CC.
>>
>> #syz dup: WARNING: refcount bug in sk_alloc
>>
>
> Excuse me? This "refcount bug in should_fail" is talking about sget_userns() versus rpc_fill_super().
> I think we need to fix either 9ee332d99e4d5a97 or d91ee87d8d85a080.
Hi,
I think I was looking at this incarnation of this bug before marking it as dup:
https://syzkaller.appspot.com/text?tag=CrashReport&id=5246446760624128
that report in fact includes sk_alloc frame. Kernel turning crash
reports into untangleable mess is not really helpful.
I will undup this into an independent bug.
It's just that we don't have such functionality yet, so I need to
implement it first.
>> >> syzbot wrote:
>> >>> Hello,
>> >>>
>> >>> syzbot hit the following crash on bpf-next commit
>> >>> 6f1b5a2b58d8470e5a8b25ab29f5fdb4616ffff8 (Tue Feb 27 04:11:23 2018 +0000)
>> >>> Merge branch 'bpf-kselftest-improvements'
>> >>>
>> >>> C reproducer is attached.
>> >>> syzkaller reproducer is attached.
>> >>> Raw console output is attached.
>> >>> compiler: gcc (GCC) 7.1.1 20170620
>> >>> .config is attached.
>> >>>
>> >>> IMPORTANT: if you fix the bug, please add the following tag to the commit:
>> >>> Reported-by: syzbot+84371b6062cb639d797e@syzkaller.appspotmail.com
>> >>> It will help syzbot understand when the bug is fixed. See footer for
>> >>> details.
>> >>> If you forward the report, please keep this part and the footer.
>> >>>
>> >>> ------------[ cut here ]------------
>> >>> FAULT_INJECTION: forcing a failure.
>> >>> name failslab, interval 1, probability 0, space 0, times 0
>> >>> refcount_t: underflow; use-after-free.
>> >>> CPU: 1 PID: 4239 Comm: syzkaller149381 Not tainted 4.16.0-rc2+ #20
>> >>> WARNING: CPU: 0 PID: 4237 at lib/refcount.c:187
>> >>> refcount_sub_and_test+0x167/0x1b0 lib/refcount.c:187
>> >>> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
>> >>> Google 01/01/2011
>> >>> Call Trace:
>> >>> Kernel panic - not syncing: panic_on_warn set ...
>> >>>
>> >>> __dump_stack lib/dump_stack.c:17 [inline]
>> >>> dump_stack+0x194/0x24d lib/dump_stack.c:53
>> >>> fail_dump lib/fault-inject.c:51 [inline]
>> >>> should_fail+0x8c0/0xa40 lib/fault-inject.c:149
>> >>> should_failslab+0xec/0x120 mm/failslab.c:32
>> >>> slab_pre_alloc_hook mm/slab.h:422 [inline]
>> >>> slab_alloc mm/slab.c:3365 [inline]
>> >>> __do_kmalloc mm/slab.c:3703 [inline]
>> >>> __kmalloc+0x63/0x760 mm/slab.c:3714
>> >>> kmalloc include/linux/slab.h:517 [inline]
>> >>> kzalloc include/linux/slab.h:701 [inline]
>> >>> register_shrinker+0x10e/0x2d0 mm/vmscan.c:268
>> >>> sget_userns+0xbbf/0xe40 fs/super.c:520
>> >>> mount_ns+0x6d/0x190 fs/super.c:1029
>> >>> rpc_mount+0x9e/0xd0 net/sunrpc/rpc_pipe.c:1451
>> >>> mount_fs+0x66/0x2d0 fs/super.c:1222
>> >>> vfs_kern_mount.part.26+0xc6/0x4a0 fs/namespace.c:1037
>> >>> vfs_kern_mount fs/namespace.c:2509 [inline]
>> >>> do_new_mount fs/namespace.c:2512 [inline]
>> >>> do_mount+0xea4/0x2bb0 fs/namespace.c:2842
>> >>> SYSC_mount fs/namespace.c:3058 [inline]
>> >>> SyS_mount+0xab/0x120 fs/namespace.c:3035
>> >>> do_syscall_64+0x280/0x940 arch/x86/entry/common.c:287
>> >>> entry_SYSCALL_64_after_hwframe+0x42/0xb7
>> >>> RIP: 0033:0x4460f9
>> >>> RSP: 002b:00007fbcd769ad78 EFLAGS: 00000246 ORIG_RAX: 00000000000000a5
>> >>> RAX: ffffffffffffffda RBX: 00000000006dcc6c RCX: 00000000004460f9
>> >>> RDX: 0000000020000080 RSI: 0000000020000040 RDI: 0000000020000000
>> >>> RBP: 00007fbcd769ad80 R08: 00000000200000c0 R09: 0000000000003131
>> >>> R10: 0000000000000000 R11: 0000000000000246 R12: 00000000006dcc68
>> >>> R13: ffffffffffffffff R14: 0000000000000037 R15: 0030656c69662f2e
>> >>> CPU: 0 PID: 4237 Comm: syzkaller149381 Not tainted 4.16.0-rc2+ #20
>> >>> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
>> >>> Google 01/01/2011
>> >>> Call Trace:
>> >>> __dump_stack lib/dump_stack.c:17 [inline]
>> >>> dump_stack+0x194/0x24d lib/dump_stack.c:53
>> >>> panic+0x1e4/0x41c kernel/panic.c:183
>> >>> __warn+0x1dc/0x200 kernel/panic.c:547
>> >>> report_bug+0x211/0x2d0 lib/bug.c:184
>> >>> fixup_bug.part.11+0x37/0x80 arch/x86/kernel/traps.c:178
>> >>> fixup_bug arch/x86/kernel/traps.c:247 [inline]
>> >>> do_error_trap+0x2d7/0x3e0 arch/x86/kernel/traps.c:296
>> >>> do_invalid_op+0x1b/0x20 arch/x86/kernel/traps.c:315
>> >>> invalid_op+0x58/0x80 arch/x86/entry/entry_64.S:957
>> >>> RIP: 0010:refcount_sub_and_test+0x167/0x1b0 lib/refcount.c:187
>> >>> RSP: 0018:ffff8801b164f6d8 EFLAGS: 00010286
>> >>> RAX: dffffc0000000008 RBX: 0000000000000000 RCX: ffffffff815ac30e
>> >>> RDX: 0000000000000000 RSI: 1ffff100362c9e8b RDI: 1ffff100362c9e60
>> >>> RBP: ffff8801b164f768 R08: 0000000000000000 R09: 0000000000000000
>> >>> R10: ffff8801b164f610 R11: 0000000000000000 R12: 1ffff100362c9edc
>> >>> R13: 00000000ffffffff R14: 0000000000000001 R15: ffff8801ae924044
>> >>> refcount_dec_and_test+0x1a/0x20 lib/refcount.c:212
>> >>> put_net include/net/net_namespace.h:220 [inline]
>> >>> rpc_kill_sb+0x253/0x3c0 net/sunrpc/rpc_pipe.c:1473
>> >>> deactivate_locked_super+0x88/0xd0 fs/super.c:312
>> >>> sget_userns+0xbda/0xe40 fs/super.c:522
>> >>> mount_ns+0x6d/0x190 fs/super.c:1029
>> >>> rpc_mount+0x9e/0xd0 net/sunrpc/rpc_pipe.c:1451
>> >>> mount_fs+0x66/0x2d0 fs/super.c:1222
>> >>> vfs_kern_mount.part.26+0xc6/0x4a0 fs/namespace.c:1037
>> >>> vfs_kern_mount fs/namespace.c:2509 [inline]
>> >>> do_new_mount fs/namespace.c:2512 [inline]
>> >>> do_mount+0xea4/0x2bb0 fs/namespace.c:2842
>> >>> SYSC_mount fs/namespace.c:3058 [inline]
>> >>> SyS_mount+0xab/0x120 fs/namespace.c:3035
>> >>> do_syscall_64+0x280/0x940 arch/x86/entry/common.c:287
>> >>> entry_SYSCALL_64_after_hwframe+0x42/0xb7
>> >>> RIP: 0033:0x4460f9
>> >>> RSP: 002b:00007fbcd76dcd78 EFLAGS: 00000246 ORIG_RAX: 00000000000000a5
>> >>> RAX: ffffffffffffffda RBX: 00000000006dcc3c RCX: 00000000004460f9
>> >>> RDX: 0000000020000080 RSI: 0000000020000040 RDI: 0000000020000000
>> >>> RBP: 00007fbcd76dcd80 R08: 00000000200000c0 R09: 0000000000003131
>> >>> R10: 0000000000000000 R11: 0000000000000246 R12: 00000000006dcc38
>> >>> R13: ffffffffffffffff R14: 0000000000000028 R15: 0030656c69662f2e
>> >>> Dumping ftrace buffer:
>> >>> (ftrace buffer empty)
>> >>> Kernel Offset: disabled
>> >>> Rebooting in 86400 seconds..
>> >>>
>> >>>
>> >>> ---
>> >>> This bug is generated by a dumb bot. It may contain errors.
>> >>> See https://goo.gl/tpsmEJ for details.
>> >>> Direct all questions to syzkaller@googlegroups.com.
>> >>>
>> >>> syzbot will keep track of this bug report.
>> >>> If you forgot to add the Reported-by tag, once the fix for this bug is
>> >>> merged
>> >>> into any tree, please reply to this email with:
>> >>> #syz fix: exact-commit-title
>> >>> If you want to test a patch for this bug, please reply with:
>> >>> #syz test: git://repo/address.git branch
>> >>> and provide the patch inline or as an attachment.
>> >>> To mark this as a duplicate of another syzbot report, please reply with:
>> >>> #syz dup: exact-subject-of-another-report
>> >>> If it's a one-off invalid bug report, please reply with:
>> >>> #syz invalid
>> >>> Note: if the crash happens again, it will cause creation of a new bug
>> >>> report.
>> >>> Note: all commands must start from beginning of the line in the email body.
>> >>>
>> >>
>> >> --
>> >> You received this message because you are subscribed to the Google Groups "syzkaller-bugs" group.
>> >> To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller-bugs+unsubscribe@googlegroups.com.
>> >> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller-bugs/201803041457.GBJ69774.OVOSOLFHQMJFFt%40I-love.SAKURA.ne.jp.
>> >> For more options, visit https://groups.google.com/d/optout.
>>
>
> --
> You received this message because you are subscribed to the Google Groups "syzkaller-bugs" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller-bugs+unsubscribe@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller-bugs/201804012011.ABI74044.MJtOFVSHFQFOOL%40I-love.SAKURA.ne.jp.
> For more options, visit https://groups.google.com/d/optout.
^ permalink raw reply
* Re: WARNING: refcount bug in should_fail
From: Dmitry Vyukov @ 2018-04-01 11:46 UTC (permalink / raw)
To: Tetsuo Handa
Cc: Al Viro, Eric W. Biederman, linux-fsdevel, netdev, Linux-MM,
syzkaller-bugs, syzbot+84371b6062cb639d797e
In-Reply-To: <CACT4Y+YHabtuOKDUK9454pX+tpdze3iSuArFuEpgOQzriq6w4w@mail.gmail.com>
On Sun, Apr 1, 2018 at 1:30 PM, Dmitry Vyukov <dvyukov@google.com> wrote:
> On Sun, Apr 1, 2018 at 1:11 PM, Tetsuo Handa
> <penguin-kernel@i-love.sakura.ne.jp> wrote:
>> Dmitry Vyukov wrote:
>>> On Sun, Apr 1, 2018 at 12:32 PM, Dmitry Vyukov <dvyukov@google.com> wrote:
>>> > On Sun, Mar 4, 2018 at 6:57 AM, Tetsuo Handa
>>> > <penguin-kernel@i-love.sakura.ne.jp> wrote:
>>> >> Switching from mm to fsdevel, for this report says that put_net(net) in
>>> >> rpc_kill_sb() made net->count < 0 when mount_ns() failed due to
>>> >> register_shrinker() failure.
>>> >>
>>> >> Relevant commits will be
>>> >> commit 9ee332d99e4d5a97 ("sget(): handle failures of register_shrinker()") and
>>> >> commit d91ee87d8d85a080 ("vfs: Pass data, ns, and ns->userns to mount_ns.").
>>> >>
>>> >> When sget_userns() in mount_ns() failed, mount_ns() returns an error code to
>>> >> the caller without calling fill_super(). That is, get_net(sb->s_fs_info) was
>>> >> not called by rpc_fill_super() (via fill_super callback passed to mount_ns())
>>> >> but put_net(sb->s_fs_info) is called by rpc_kill_sb() (via fs->kill_sb() from
>>> >> deactivate_locked_super()).
>>> >>
>>> >> ----------
>>> >> static struct dentry *
>>> >> rpc_mount(struct file_system_type *fs_type,
>>> >> int flags, const char *dev_name, void *data)
>>> >> {
>>> >> struct net *net = current->nsproxy->net_ns;
>>> >> return mount_ns(fs_type, flags, data, net, net->user_ns, rpc_fill_super);
>>> >> }
>>> >> ----------
>>> >
>>> > Messed kernel output, this is definitely not in should_fail.
>>> >
>>> > #syz dup: WARNING: refcount bug in sk_alloc
>>>
>>> Please don't drop reporter (syzbot) email from CC.
>>>
>>> #syz dup: WARNING: refcount bug in sk_alloc
>>>
>>
>> Excuse me? This "refcount bug in should_fail" is talking about sget_userns() versus rpc_fill_super().
>> I think we need to fix either 9ee332d99e4d5a97 or d91ee87d8d85a080.
>
> Hi,
>
> I think I was looking at this incarnation of this bug before marking it as dup:
> https://syzkaller.appspot.com/text?tag=CrashReport&id=5246446760624128
> that report in fact includes sk_alloc frame. Kernel turning crash
> reports into untangleable mess is not really helpful.
>
> I will undup this into an independent bug.
> It's just that we don't have such functionality yet, so I need to
> implement it first.
This is now implemented:
https://github.com/google/syzkaller/commit/0a78e248b7b6537ccdf66dc8806d76e0a97efe21
Let's try it:
#syz undup
>>> >> syzbot wrote:
>>> >>> Hello,
>>> >>>
>>> >>> syzbot hit the following crash on bpf-next commit
>>> >>> 6f1b5a2b58d8470e5a8b25ab29f5fdb4616ffff8 (Tue Feb 27 04:11:23 2018 +0000)
>>> >>> Merge branch 'bpf-kselftest-improvements'
>>> >>>
>>> >>> C reproducer is attached.
>>> >>> syzkaller reproducer is attached.
>>> >>> Raw console output is attached.
>>> >>> compiler: gcc (GCC) 7.1.1 20170620
>>> >>> .config is attached.
>>> >>>
>>> >>> IMPORTANT: if you fix the bug, please add the following tag to the commit:
>>> >>> Reported-by: syzbot+84371b6062cb639d797e@syzkaller.appspotmail.com
>>> >>> It will help syzbot understand when the bug is fixed. See footer for
>>> >>> details.
>>> >>> If you forward the report, please keep this part and the footer.
>>> >>>
>>> >>> ------------[ cut here ]------------
>>> >>> FAULT_INJECTION: forcing a failure.
>>> >>> name failslab, interval 1, probability 0, space 0, times 0
>>> >>> refcount_t: underflow; use-after-free.
>>> >>> CPU: 1 PID: 4239 Comm: syzkaller149381 Not tainted 4.16.0-rc2+ #20
>>> >>> WARNING: CPU: 0 PID: 4237 at lib/refcount.c:187
>>> >>> refcount_sub_and_test+0x167/0x1b0 lib/refcount.c:187
>>> >>> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
>>> >>> Google 01/01/2011
>>> >>> Call Trace:
>>> >>> Kernel panic - not syncing: panic_on_warn set ...
>>> >>>
>>> >>> __dump_stack lib/dump_stack.c:17 [inline]
>>> >>> dump_stack+0x194/0x24d lib/dump_stack.c:53
>>> >>> fail_dump lib/fault-inject.c:51 [inline]
>>> >>> should_fail+0x8c0/0xa40 lib/fault-inject.c:149
>>> >>> should_failslab+0xec/0x120 mm/failslab.c:32
>>> >>> slab_pre_alloc_hook mm/slab.h:422 [inline]
>>> >>> slab_alloc mm/slab.c:3365 [inline]
>>> >>> __do_kmalloc mm/slab.c:3703 [inline]
>>> >>> __kmalloc+0x63/0x760 mm/slab.c:3714
>>> >>> kmalloc include/linux/slab.h:517 [inline]
>>> >>> kzalloc include/linux/slab.h:701 [inline]
>>> >>> register_shrinker+0x10e/0x2d0 mm/vmscan.c:268
>>> >>> sget_userns+0xbbf/0xe40 fs/super.c:520
>>> >>> mount_ns+0x6d/0x190 fs/super.c:1029
>>> >>> rpc_mount+0x9e/0xd0 net/sunrpc/rpc_pipe.c:1451
>>> >>> mount_fs+0x66/0x2d0 fs/super.c:1222
>>> >>> vfs_kern_mount.part.26+0xc6/0x4a0 fs/namespace.c:1037
>>> >>> vfs_kern_mount fs/namespace.c:2509 [inline]
>>> >>> do_new_mount fs/namespace.c:2512 [inline]
>>> >>> do_mount+0xea4/0x2bb0 fs/namespace.c:2842
>>> >>> SYSC_mount fs/namespace.c:3058 [inline]
>>> >>> SyS_mount+0xab/0x120 fs/namespace.c:3035
>>> >>> do_syscall_64+0x280/0x940 arch/x86/entry/common.c:287
>>> >>> entry_SYSCALL_64_after_hwframe+0x42/0xb7
>>> >>> RIP: 0033:0x4460f9
>>> >>> RSP: 002b:00007fbcd769ad78 EFLAGS: 00000246 ORIG_RAX: 00000000000000a5
>>> >>> RAX: ffffffffffffffda RBX: 00000000006dcc6c RCX: 00000000004460f9
>>> >>> RDX: 0000000020000080 RSI: 0000000020000040 RDI: 0000000020000000
>>> >>> RBP: 00007fbcd769ad80 R08: 00000000200000c0 R09: 0000000000003131
>>> >>> R10: 0000000000000000 R11: 0000000000000246 R12: 00000000006dcc68
>>> >>> R13: ffffffffffffffff R14: 0000000000000037 R15: 0030656c69662f2e
>>> >>> CPU: 0 PID: 4237 Comm: syzkaller149381 Not tainted 4.16.0-rc2+ #20
>>> >>> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
>>> >>> Google 01/01/2011
>>> >>> Call Trace:
>>> >>> __dump_stack lib/dump_stack.c:17 [inline]
>>> >>> dump_stack+0x194/0x24d lib/dump_stack.c:53
>>> >>> panic+0x1e4/0x41c kernel/panic.c:183
>>> >>> __warn+0x1dc/0x200 kernel/panic.c:547
>>> >>> report_bug+0x211/0x2d0 lib/bug.c:184
>>> >>> fixup_bug.part.11+0x37/0x80 arch/x86/kernel/traps.c:178
>>> >>> fixup_bug arch/x86/kernel/traps.c:247 [inline]
>>> >>> do_error_trap+0x2d7/0x3e0 arch/x86/kernel/traps.c:296
>>> >>> do_invalid_op+0x1b/0x20 arch/x86/kernel/traps.c:315
>>> >>> invalid_op+0x58/0x80 arch/x86/entry/entry_64.S:957
>>> >>> RIP: 0010:refcount_sub_and_test+0x167/0x1b0 lib/refcount.c:187
>>> >>> RSP: 0018:ffff8801b164f6d8 EFLAGS: 00010286
>>> >>> RAX: dffffc0000000008 RBX: 0000000000000000 RCX: ffffffff815ac30e
>>> >>> RDX: 0000000000000000 RSI: 1ffff100362c9e8b RDI: 1ffff100362c9e60
>>> >>> RBP: ffff8801b164f768 R08: 0000000000000000 R09: 0000000000000000
>>> >>> R10: ffff8801b164f610 R11: 0000000000000000 R12: 1ffff100362c9edc
>>> >>> R13: 00000000ffffffff R14: 0000000000000001 R15: ffff8801ae924044
>>> >>> refcount_dec_and_test+0x1a/0x20 lib/refcount.c:212
>>> >>> put_net include/net/net_namespace.h:220 [inline]
>>> >>> rpc_kill_sb+0x253/0x3c0 net/sunrpc/rpc_pipe.c:1473
>>> >>> deactivate_locked_super+0x88/0xd0 fs/super.c:312
>>> >>> sget_userns+0xbda/0xe40 fs/super.c:522
>>> >>> mount_ns+0x6d/0x190 fs/super.c:1029
>>> >>> rpc_mount+0x9e/0xd0 net/sunrpc/rpc_pipe.c:1451
>>> >>> mount_fs+0x66/0x2d0 fs/super.c:1222
>>> >>> vfs_kern_mount.part.26+0xc6/0x4a0 fs/namespace.c:1037
>>> >>> vfs_kern_mount fs/namespace.c:2509 [inline]
>>> >>> do_new_mount fs/namespace.c:2512 [inline]
>>> >>> do_mount+0xea4/0x2bb0 fs/namespace.c:2842
>>> >>> SYSC_mount fs/namespace.c:3058 [inline]
>>> >>> SyS_mount+0xab/0x120 fs/namespace.c:3035
>>> >>> do_syscall_64+0x280/0x940 arch/x86/entry/common.c:287
>>> >>> entry_SYSCALL_64_after_hwframe+0x42/0xb7
>>> >>> RIP: 0033:0x4460f9
>>> >>> RSP: 002b:00007fbcd76dcd78 EFLAGS: 00000246 ORIG_RAX: 00000000000000a5
>>> >>> RAX: ffffffffffffffda RBX: 00000000006dcc3c RCX: 00000000004460f9
>>> >>> RDX: 0000000020000080 RSI: 0000000020000040 RDI: 0000000020000000
>>> >>> RBP: 00007fbcd76dcd80 R08: 00000000200000c0 R09: 0000000000003131
>>> >>> R10: 0000000000000000 R11: 0000000000000246 R12: 00000000006dcc38
>>> >>> R13: ffffffffffffffff R14: 0000000000000028 R15: 0030656c69662f2e
>>> >>> Dumping ftrace buffer:
>>> >>> (ftrace buffer empty)
>>> >>> Kernel Offset: disabled
>>> >>> Rebooting in 86400 seconds..
>>> >>>
>>> >>>
>>> >>> ---
>>> >>> This bug is generated by a dumb bot. It may contain errors.
>>> >>> See https://goo.gl/tpsmEJ for details.
>>> >>> Direct all questions to syzkaller@googlegroups.com.
>>> >>>
>>> >>> syzbot will keep track of this bug report.
>>> >>> If you forgot to add the Reported-by tag, once the fix for this bug is
>>> >>> merged
>>> >>> into any tree, please reply to this email with:
>>> >>> #syz fix: exact-commit-title
>>> >>> If you want to test a patch for this bug, please reply with:
>>> >>> #syz test: git://repo/address.git branch
>>> >>> and provide the patch inline or as an attachment.
>>> >>> To mark this as a duplicate of another syzbot report, please reply with:
>>> >>> #syz dup: exact-subject-of-another-report
>>> >>> If it's a one-off invalid bug report, please reply with:
>>> >>> #syz invalid
>>> >>> Note: if the crash happens again, it will cause creation of a new bug
>>> >>> report.
>>> >>> Note: all commands must start from beginning of the line in the email body.
>>> >>>
>>> >>
>>> >> --
>>> >> You received this message because you are subscribed to the Google Groups "syzkaller-bugs" group.
>>> >> To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller-bugs+unsubscribe@googlegroups.com.
>>> >> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller-bugs/201803041457.GBJ69774.OVOSOLFHQMJFFt%40I-love.SAKURA.ne.jp.
>>> >> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups "syzkaller-bugs" group.
>> To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller-bugs+unsubscribe@googlegroups.com.
>> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller-bugs/201804012011.ABI74044.MJtOFVSHFQFOOL%40I-love.SAKURA.ne.jp.
>> For more options, visit https://groups.google.com/d/optout.
^ permalink raw reply
* Re: WARNING: refcount bug in should_fail
From: Dmitry Vyukov @ 2018-04-01 11:50 UTC (permalink / raw)
To: Tetsuo Handa
Cc: Al Viro, Eric W. Biederman, linux-fsdevel, netdev, Linux-MM,
syzkaller-bugs, syzbot+84371b6062cb639d797e
In-Reply-To: <CACT4Y+b-Mety+eWj12=uCPPfo6iK=P4AeAb1Hxa8Ltci-cZmkQ@mail.gmail.com>
On Sun, Apr 1, 2018 at 1:46 PM, Dmitry Vyukov <dvyukov@google.com> wrote:
> On Sun, Apr 1, 2018 at 1:30 PM, Dmitry Vyukov <dvyukov@google.com> wrote:
>> On Sun, Apr 1, 2018 at 1:11 PM, Tetsuo Handa
>> <penguin-kernel@i-love.sakura.ne.jp> wrote:
>>> Dmitry Vyukov wrote:
>>>> On Sun, Apr 1, 2018 at 12:32 PM, Dmitry Vyukov <dvyukov@google.com> wrote:
>>>> > On Sun, Mar 4, 2018 at 6:57 AM, Tetsuo Handa
>>>> > <penguin-kernel@i-love.sakura.ne.jp> wrote:
>>>> >> Switching from mm to fsdevel, for this report says that put_net(net) in
>>>> >> rpc_kill_sb() made net->count < 0 when mount_ns() failed due to
>>>> >> register_shrinker() failure.
>>>> >>
>>>> >> Relevant commits will be
>>>> >> commit 9ee332d99e4d5a97 ("sget(): handle failures of register_shrinker()") and
>>>> >> commit d91ee87d8d85a080 ("vfs: Pass data, ns, and ns->userns to mount_ns.").
>>>> >>
>>>> >> When sget_userns() in mount_ns() failed, mount_ns() returns an error code to
>>>> >> the caller without calling fill_super(). That is, get_net(sb->s_fs_info) was
>>>> >> not called by rpc_fill_super() (via fill_super callback passed to mount_ns())
>>>> >> but put_net(sb->s_fs_info) is called by rpc_kill_sb() (via fs->kill_sb() from
>>>> >> deactivate_locked_super()).
>>>> >>
>>>> >> ----------
>>>> >> static struct dentry *
>>>> >> rpc_mount(struct file_system_type *fs_type,
>>>> >> int flags, const char *dev_name, void *data)
>>>> >> {
>>>> >> struct net *net = current->nsproxy->net_ns;
>>>> >> return mount_ns(fs_type, flags, data, net, net->user_ns, rpc_fill_super);
>>>> >> }
>>>> >> ----------
>>>> >
>>>> > Messed kernel output, this is definitely not in should_fail.
>>>> >
>>>> > #syz dup: WARNING: refcount bug in sk_alloc
>>>>
>>>> Please don't drop reporter (syzbot) email from CC.
>>>>
>>>> #syz dup: WARNING: refcount bug in sk_alloc
>>>>
>>>
>>> Excuse me? This "refcount bug in should_fail" is talking about sget_userns() versus rpc_fill_super().
>>> I think we need to fix either 9ee332d99e4d5a97 or d91ee87d8d85a080.
>>
>> Hi,
>>
>> I think I was looking at this incarnation of this bug before marking it as dup:
>> https://syzkaller.appspot.com/text?tag=CrashReport&id=5246446760624128
>> that report in fact includes sk_alloc frame. Kernel turning crash
>> reports into untangleable mess is not really helpful.
>>
>> I will undup this into an independent bug.
>> It's just that we don't have such functionality yet, so I need to
>> implement it first.
>
> This is now implemented:
> https://github.com/google/syzkaller/commit/0a78e248b7b6537ccdf66dc8806d76e0a97efe21
> Let's try it:
>
> #syz undup
OK, this is now a separate bug again:
https://syzkaller.appspot.com/bug?id=55c7257f74dd17f65a9d057b316c46d156b0fba4
The last step is that somebody needs to actually fix it (until too
many different bugs pile up under "WARNING: refcount bug in
should_fail") ;)
>>>> >> syzbot wrote:
>>>> >>> Hello,
>>>> >>>
>>>> >>> syzbot hit the following crash on bpf-next commit
>>>> >>> 6f1b5a2b58d8470e5a8b25ab29f5fdb4616ffff8 (Tue Feb 27 04:11:23 2018 +0000)
>>>> >>> Merge branch 'bpf-kselftest-improvements'
>>>> >>>
>>>> >>> C reproducer is attached.
>>>> >>> syzkaller reproducer is attached.
>>>> >>> Raw console output is attached.
>>>> >>> compiler: gcc (GCC) 7.1.1 20170620
>>>> >>> .config is attached.
>>>> >>>
>>>> >>> IMPORTANT: if you fix the bug, please add the following tag to the commit:
>>>> >>> Reported-by: syzbot+84371b6062cb639d797e@syzkaller.appspotmail.com
>>>> >>> It will help syzbot understand when the bug is fixed. See footer for
>>>> >>> details.
>>>> >>> If you forward the report, please keep this part and the footer.
>>>> >>>
>>>> >>> ------------[ cut here ]------------
>>>> >>> FAULT_INJECTION: forcing a failure.
>>>> >>> name failslab, interval 1, probability 0, space 0, times 0
>>>> >>> refcount_t: underflow; use-after-free.
>>>> >>> CPU: 1 PID: 4239 Comm: syzkaller149381 Not tainted 4.16.0-rc2+ #20
>>>> >>> WARNING: CPU: 0 PID: 4237 at lib/refcount.c:187
>>>> >>> refcount_sub_and_test+0x167/0x1b0 lib/refcount.c:187
>>>> >>> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
>>>> >>> Google 01/01/2011
>>>> >>> Call Trace:
>>>> >>> Kernel panic - not syncing: panic_on_warn set ...
>>>> >>>
>>>> >>> __dump_stack lib/dump_stack.c:17 [inline]
>>>> >>> dump_stack+0x194/0x24d lib/dump_stack.c:53
>>>> >>> fail_dump lib/fault-inject.c:51 [inline]
>>>> >>> should_fail+0x8c0/0xa40 lib/fault-inject.c:149
>>>> >>> should_failslab+0xec/0x120 mm/failslab.c:32
>>>> >>> slab_pre_alloc_hook mm/slab.h:422 [inline]
>>>> >>> slab_alloc mm/slab.c:3365 [inline]
>>>> >>> __do_kmalloc mm/slab.c:3703 [inline]
>>>> >>> __kmalloc+0x63/0x760 mm/slab.c:3714
>>>> >>> kmalloc include/linux/slab.h:517 [inline]
>>>> >>> kzalloc include/linux/slab.h:701 [inline]
>>>> >>> register_shrinker+0x10e/0x2d0 mm/vmscan.c:268
>>>> >>> sget_userns+0xbbf/0xe40 fs/super.c:520
>>>> >>> mount_ns+0x6d/0x190 fs/super.c:1029
>>>> >>> rpc_mount+0x9e/0xd0 net/sunrpc/rpc_pipe.c:1451
>>>> >>> mount_fs+0x66/0x2d0 fs/super.c:1222
>>>> >>> vfs_kern_mount.part.26+0xc6/0x4a0 fs/namespace.c:1037
>>>> >>> vfs_kern_mount fs/namespace.c:2509 [inline]
>>>> >>> do_new_mount fs/namespace.c:2512 [inline]
>>>> >>> do_mount+0xea4/0x2bb0 fs/namespace.c:2842
>>>> >>> SYSC_mount fs/namespace.c:3058 [inline]
>>>> >>> SyS_mount+0xab/0x120 fs/namespace.c:3035
>>>> >>> do_syscall_64+0x280/0x940 arch/x86/entry/common.c:287
>>>> >>> entry_SYSCALL_64_after_hwframe+0x42/0xb7
>>>> >>> RIP: 0033:0x4460f9
>>>> >>> RSP: 002b:00007fbcd769ad78 EFLAGS: 00000246 ORIG_RAX: 00000000000000a5
>>>> >>> RAX: ffffffffffffffda RBX: 00000000006dcc6c RCX: 00000000004460f9
>>>> >>> RDX: 0000000020000080 RSI: 0000000020000040 RDI: 0000000020000000
>>>> >>> RBP: 00007fbcd769ad80 R08: 00000000200000c0 R09: 0000000000003131
>>>> >>> R10: 0000000000000000 R11: 0000000000000246 R12: 00000000006dcc68
>>>> >>> R13: ffffffffffffffff R14: 0000000000000037 R15: 0030656c69662f2e
>>>> >>> CPU: 0 PID: 4237 Comm: syzkaller149381 Not tainted 4.16.0-rc2+ #20
>>>> >>> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
>>>> >>> Google 01/01/2011
>>>> >>> Call Trace:
>>>> >>> __dump_stack lib/dump_stack.c:17 [inline]
>>>> >>> dump_stack+0x194/0x24d lib/dump_stack.c:53
>>>> >>> panic+0x1e4/0x41c kernel/panic.c:183
>>>> >>> __warn+0x1dc/0x200 kernel/panic.c:547
>>>> >>> report_bug+0x211/0x2d0 lib/bug.c:184
>>>> >>> fixup_bug.part.11+0x37/0x80 arch/x86/kernel/traps.c:178
>>>> >>> fixup_bug arch/x86/kernel/traps.c:247 [inline]
>>>> >>> do_error_trap+0x2d7/0x3e0 arch/x86/kernel/traps.c:296
>>>> >>> do_invalid_op+0x1b/0x20 arch/x86/kernel/traps.c:315
>>>> >>> invalid_op+0x58/0x80 arch/x86/entry/entry_64.S:957
>>>> >>> RIP: 0010:refcount_sub_and_test+0x167/0x1b0 lib/refcount.c:187
>>>> >>> RSP: 0018:ffff8801b164f6d8 EFLAGS: 00010286
>>>> >>> RAX: dffffc0000000008 RBX: 0000000000000000 RCX: ffffffff815ac30e
>>>> >>> RDX: 0000000000000000 RSI: 1ffff100362c9e8b RDI: 1ffff100362c9e60
>>>> >>> RBP: ffff8801b164f768 R08: 0000000000000000 R09: 0000000000000000
>>>> >>> R10: ffff8801b164f610 R11: 0000000000000000 R12: 1ffff100362c9edc
>>>> >>> R13: 00000000ffffffff R14: 0000000000000001 R15: ffff8801ae924044
>>>> >>> refcount_dec_and_test+0x1a/0x20 lib/refcount.c:212
>>>> >>> put_net include/net/net_namespace.h:220 [inline]
>>>> >>> rpc_kill_sb+0x253/0x3c0 net/sunrpc/rpc_pipe.c:1473
>>>> >>> deactivate_locked_super+0x88/0xd0 fs/super.c:312
>>>> >>> sget_userns+0xbda/0xe40 fs/super.c:522
>>>> >>> mount_ns+0x6d/0x190 fs/super.c:1029
>>>> >>> rpc_mount+0x9e/0xd0 net/sunrpc/rpc_pipe.c:1451
>>>> >>> mount_fs+0x66/0x2d0 fs/super.c:1222
>>>> >>> vfs_kern_mount.part.26+0xc6/0x4a0 fs/namespace.c:1037
>>>> >>> vfs_kern_mount fs/namespace.c:2509 [inline]
>>>> >>> do_new_mount fs/namespace.c:2512 [inline]
>>>> >>> do_mount+0xea4/0x2bb0 fs/namespace.c:2842
>>>> >>> SYSC_mount fs/namespace.c:3058 [inline]
>>>> >>> SyS_mount+0xab/0x120 fs/namespace.c:3035
>>>> >>> do_syscall_64+0x280/0x940 arch/x86/entry/common.c:287
>>>> >>> entry_SYSCALL_64_after_hwframe+0x42/0xb7
>>>> >>> RIP: 0033:0x4460f9
>>>> >>> RSP: 002b:00007fbcd76dcd78 EFLAGS: 00000246 ORIG_RAX: 00000000000000a5
>>>> >>> RAX: ffffffffffffffda RBX: 00000000006dcc3c RCX: 00000000004460f9
>>>> >>> RDX: 0000000020000080 RSI: 0000000020000040 RDI: 0000000020000000
>>>> >>> RBP: 00007fbcd76dcd80 R08: 00000000200000c0 R09: 0000000000003131
>>>> >>> R10: 0000000000000000 R11: 0000000000000246 R12: 00000000006dcc38
>>>> >>> R13: ffffffffffffffff R14: 0000000000000028 R15: 0030656c69662f2e
>>>> >>> Dumping ftrace buffer:
>>>> >>> (ftrace buffer empty)
>>>> >>> Kernel Offset: disabled
>>>> >>> Rebooting in 86400 seconds..
>>>> >>>
>>>> >>>
>>>> >>> ---
>>>> >>> This bug is generated by a dumb bot. It may contain errors.
>>>> >>> See https://goo.gl/tpsmEJ for details.
>>>> >>> Direct all questions to syzkaller@googlegroups.com.
>>>> >>>
>>>> >>> syzbot will keep track of this bug report.
>>>> >>> If you forgot to add the Reported-by tag, once the fix for this bug is
>>>> >>> merged
>>>> >>> into any tree, please reply to this email with:
>>>> >>> #syz fix: exact-commit-title
>>>> >>> If you want to test a patch for this bug, please reply with:
>>>> >>> #syz test: git://repo/address.git branch
>>>> >>> and provide the patch inline or as an attachment.
>>>> >>> To mark this as a duplicate of another syzbot report, please reply with:
>>>> >>> #syz dup: exact-subject-of-another-report
>>>> >>> If it's a one-off invalid bug report, please reply with:
>>>> >>> #syz invalid
>>>> >>> Note: if the crash happens again, it will cause creation of a new bug
>>>> >>> report.
>>>> >>> Note: all commands must start from beginning of the line in the email body.
>>>> >>>
>>>> >>
>>>> >> --
>>>> >> You received this message because you are subscribed to the Google Groups "syzkaller-bugs" group.
>>>> >> To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller-bugs+unsubscribe@googlegroups.com.
>>>> >> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller-bugs/201803041457.GBJ69774.OVOSOLFHQMJFFt%40I-love.SAKURA.ne.jp.
>>>> >> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups "syzkaller-bugs" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller-bugs+unsubscribe@googlegroups.com.
>>> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller-bugs/201804012011.ABI74044.MJtOFVSHFQFOOL%40I-love.SAKURA.ne.jp.
>>> For more options, visit https://groups.google.com/d/optout.
^ permalink raw reply
* Re: [PATCH v15 net-next 00/12] Chelsio Inline TLS
From: Boris Pismenny @ 2018-04-01 12:57 UTC (permalink / raw)
To: David Miller, atul.gupta
Cc: herbert, davejwatson, sd, sbrivio, linux-crypto, netdev, werner,
leedom, swise, indranil, ganeshgr
In-Reply-To: <20180331.233753.2141475779224683361.davem@davemloft.net>
Hi,
On 4/1/2018 6:37 AM, David Miller wrote:
> From: Atul Gupta <atul.gupta@chelsio.com>
> Date: Sat, 31 Mar 2018 21:41:51 +0530
>
>> Series for Chelsio Inline TLS driver (chtls)
>
> Series applied, thank you.
>
Sorry for being late to the party, could you please help answer a few
questions to help me understand better.
1. What happens if someone attempts to set a TCP socket option for a
socket whose TCP stack resides in the TCP offload engine(TOE)? Do you
emulate all Linux socket options? What about IP socket options?
If I follow the code correctly, then the original TCP/IP setsockopt is
called. But, it doesn't change any of the parameters of the TCP/IP
offload engine in hardware.
2. I can't find where is the TLS record sequence number pushed to
hardware. Is that on purpose?
FYI, ignoring this parameter might cause a record sequence number reuse
which breaks the integrity of the AES-GCM TLS ciphersuite.
3. How does a TOE handle Tx only or Rx only?
4. What happens when there is a routing change that redirects traffic to
a different netdev? Is there a software fallback?
5. The TLS socket option is set in the middle of a TCP connection. What
happens to the existing TCP connection and the data that is currently
queued in the TCP write queue?
Thanks,
Boris.
^ permalink raw reply
* [PATCH net-next] net: ipv6/gre: Add GRO support
From: Tariq Toukan @ 2018-04-01 13:17 UTC (permalink / raw)
To: David S. Miller; +Cc: netdev, Eran Ben Elisha, Tariq Toukan, Eric Dumazet
From: Eran Ben Elisha <eranbe@mellanox.com>
Add GRO capability for IPv6 GRE tunnel and ip6erspan tap, via gro_cells
infrastructure.
Performance testing: 55% higher badwidth.
Measuring bandwidth of 1 thread IPv4 TCP traffic over IPv6 GRE tunnel
while GRO on the physical interface is disabled.
CPU: Intel Xeon E312xx (Sandy Bridge)
NIC: Mellanox Technologies MT27700 Family [ConnectX-4]
Before (GRO not working in tunnel) : 2.47 Gbits/sec
After (GRO working in tunnel) : 3.85 Gbits/sec
Signed-off-by: Eran Ben Elisha <eranbe@mellanox.com>
Signed-off-by: Tariq Toukan <tariqt@mellanox.com>
CC: Eric Dumazet <edumazet@google.com>
---
net/ipv6/ip6_gre.c | 37 +++++++++++++++++++++++++++----------
1 file changed, 27 insertions(+), 10 deletions(-)
diff --git a/net/ipv6/ip6_gre.c b/net/ipv6/ip6_gre.c
index 22e86557aca4..8abc9d72993e 100644
--- a/net/ipv6/ip6_gre.c
+++ b/net/ipv6/ip6_gre.c
@@ -1306,6 +1306,7 @@ static void ip6gre_dev_free(struct net_device *dev)
{
struct ip6_tnl *t = netdev_priv(dev);
+ gro_cells_destroy(&t->gro_cells);
dst_cache_destroy(&t->dst_cache);
free_percpu(dev->tstats);
}
@@ -1373,11 +1374,12 @@ static int ip6gre_tunnel_init_common(struct net_device *dev)
return -ENOMEM;
ret = dst_cache_init(&tunnel->dst_cache, GFP_KERNEL);
- if (ret) {
- free_percpu(dev->tstats);
- dev->tstats = NULL;
- return ret;
- }
+ if (ret)
+ goto cleanup_alloc_pcpu_stats;
+
+ ret = gro_cells_init(&tunnel->gro_cells, dev);
+ if (ret)
+ goto cleanup_dst_cache_init;
tunnel->tun_hlen = gre_calc_hlen(tunnel->parms.o_flags);
tunnel->hlen = tunnel->tun_hlen + tunnel->encap_hlen;
@@ -1397,6 +1399,13 @@ static int ip6gre_tunnel_init_common(struct net_device *dev)
ip6gre_tnl_init_features(dev);
return 0;
+
+cleanup_dst_cache_init:
+ dst_cache_destroy(&tunnel->dst_cache);
+cleanup_alloc_pcpu_stats:
+ free_percpu(dev->tstats);
+ dev->tstats = NULL;
+ return ret;
}
static int ip6gre_tunnel_init(struct net_device *dev)
@@ -1743,11 +1752,12 @@ static int ip6erspan_tap_init(struct net_device *dev)
return -ENOMEM;
ret = dst_cache_init(&tunnel->dst_cache, GFP_KERNEL);
- if (ret) {
- free_percpu(dev->tstats);
- dev->tstats = NULL;
- return ret;
- }
+ if (ret)
+ goto cleanup_alloc_pcpu_stats;
+
+ ret = gro_cells_init(&tunnel->gro_cells, dev);
+ if (ret)
+ goto cleanup_dst_cache_init;
tunnel->tun_hlen = 8;
tunnel->hlen = tunnel->tun_hlen + tunnel->encap_hlen +
@@ -1766,6 +1776,13 @@ static int ip6erspan_tap_init(struct net_device *dev)
ip6gre_tnl_link_config(tunnel, 1);
return 0;
+
+cleanup_dst_cache_init:
+ dst_cache_destroy(&tunnel->dst_cache);
+cleanup_alloc_pcpu_stats:
+ free_percpu(dev->tstats);
+ dev->tstats = NULL;
+ return ret;
}
static const struct net_device_ops ip6erspan_netdev_ops = {
--
1.8.3.1
^ permalink raw reply related
* Re: [PATCH v15 net-next 01/12] tls: support for Inline tls record
From: Boris Pismenny @ 2018-04-01 13:22 UTC (permalink / raw)
To: Atul Gupta, davem, herbert
Cc: davejwatson, sd, sbrivio, linux-crypto, netdev, werner, leedom,
swise, indranil, ganeshgr
In-Reply-To: <1522512723-31747-2-git-send-email-atul.gupta@chelsio.com>
On 3/31/2018 7:11 PM, Atul Gupta wrote:
> Facility to register Inline TLS drivers to net/tls. Setup
> TLS_HW_RECORD prot to listen on offload device.
>
> Cases handled
> - Inline TLS device exists, setup prot for TLS_HW_RECORD
> - Atleast one Inline TLS exists, sets TLS_HW_RECORD.
> - If non-inline device establish connection, move to TLS_SW_TX
>
> Signed-off-by: Atul Gupta <atul.gupta@chelsio.com>
> Reviewed-by: Steve Wise <swise@opengridcomputing.com>
> ---
> include/net/tls.h | 32 ++++++++++++++-
> net/tls/tls_main.c | 114 +++++++++++++++++++++++++++++++++++++++++++++++++++--
> 2 files changed, 142 insertions(+), 4 deletions(-)
>
> diff --git a/include/net/tls.h b/include/net/tls.h
> index 437a746..3da8e13 100644
> --- a/include/net/tls.h
> +++ b/include/net/tls.h
> @@ -56,6 +56,32 @@
> #define TLS_RECORD_TYPE_DATA 0x17
>
> #define TLS_AAD_SPACE_SIZE 13
> +#define TLS_DEVICE_NAME_MAX 32
> +
> +/*
> + * This structure defines the routines for Inline TLS driver.
> + * The following routines are optional and filled with a
> + * null pointer if not defined.
> + *
> + * @name: Its the name of registered Inline tls device
> + * @dev_list: Inline tls device list
> + * int (*feature)(struct tls_device *device);
> + * Called to return Inline TLS driver capability
> + *
> + * int (*hash)(struct tls_device *device, struct sock *sk);
> + * This function sets Inline driver for listen and program
> + * device specific functioanlity as required
> + *
> + * void (*unhash)(struct tls_device *device, struct sock *sk);
> + * This function cleans listen state set by Inline TLS driver
> + */
> +struct tls_device {
> + char name[TLS_DEVICE_NAME_MAX];
> + struct list_head dev_list;
> + int (*feature)(struct tls_device *device);
> + int (*hash)(struct tls_device *device, struct sock *sk);
> + void (*unhash)(struct tls_device *device, struct sock *sk);
> +};
>
> struct tls_sw_context {
> struct crypto_aead *aead_send;
> @@ -114,7 +140,7 @@ struct tls_context {
>
> void *priv_ctx;
>
> - u8 conf:2;
> + u8 conf:3;
>
> struct cipher_context tx;
> struct cipher_context rx;
> @@ -135,6 +161,8 @@ struct tls_context {
> int (*getsockopt)(struct sock *sk, int level,
> int optname, char __user *optval,
> int __user *optlen);
> + int (*hash)(struct sock *sk);
> + void (*unhash)(struct sock *sk);
> };
>
> int wait_on_pending_writer(struct sock *sk, long *timeo);
> @@ -283,5 +311,7 @@ static inline struct tls_offload_context *tls_offload_ctx(
>
> int tls_proccess_cmsg(struct sock *sk, struct msghdr *msg,
> unsigned char *record_type);
> +void tls_register_device(struct tls_device *device);
> +void tls_unregister_device(struct tls_device *device);
>
> #endif /* _TLS_OFFLOAD_H */
> diff --git a/net/tls/tls_main.c b/net/tls/tls_main.c
> index 6f5c114..0d37997 100644
> --- a/net/tls/tls_main.c
> +++ b/net/tls/tls_main.c
> @@ -38,6 +38,7 @@
> #include <linux/highmem.h>
> #include <linux/netdevice.h>
> #include <linux/sched/signal.h>
> +#include <linux/inetdevice.h>
>
> #include <net/tls.h>
>
> @@ -56,11 +57,14 @@ enum {
> TLS_SW_TX,
> TLS_SW_RX,
> TLS_SW_RXTX,
> + TLS_HW_RECORD,
> TLS_NUM_CONFIG,
> };
>
> static struct proto *saved_tcpv6_prot;
> static DEFINE_MUTEX(tcpv6_prot_mutex);
> +static LIST_HEAD(device_list);
> +static DEFINE_MUTEX(device_mutex);
> static struct proto tls_prots[TLS_NUM_PROTS][TLS_NUM_CONFIG];
> static struct proto_ops tls_sw_proto_ops;
>
> @@ -241,8 +245,12 @@ static void tls_sk_proto_close(struct sock *sk, long timeout)
> lock_sock(sk);
> sk_proto_close = ctx->sk_proto_close;
>
> + if (ctx->conf == TLS_HW_RECORD)
> + goto skip_tx_cleanup;
> +
> if (ctx->conf == TLS_BASE) {
> kfree(ctx);
> + ctx = NULL;
> goto skip_tx_cleanup;
> }
>
> @@ -276,6 +284,11 @@ static void tls_sk_proto_close(struct sock *sk, long timeout)
> skip_tx_cleanup:
> release_sock(sk);
> sk_proto_close(sk, timeout);
> + /* free ctx for TLS_HW_RECORD, used by tcp_set_state
> + * for sk->sk_prot->unhash [tls_hw_unhash]
> + */
> + if (ctx && ctx->conf == TLS_HW_RECORD)
> + kfree(ctx);
> }
>
> static int do_tls_getsockopt_tx(struct sock *sk, char __user *optval,
> @@ -493,6 +506,79 @@ static int tls_setsockopt(struct sock *sk, int level, int optname,
> return do_tls_setsockopt(sk, optname, optval, optlen);
> }
>
> +static struct tls_context *create_ctx(struct sock *sk)
> +{
> + struct inet_connection_sock *icsk = inet_csk(sk);
> + struct tls_context *ctx;
> +
> + ctx = kzalloc(sizeof(*ctx), GFP_KERNEL);
> + if (!ctx)
> + return NULL;
> +
> + icsk->icsk_ulp_data = ctx;
> + return ctx;
> +}
> +
> +static int tls_hw_prot(struct sock *sk)
> +{
> + struct tls_context *ctx;
> + struct tls_device *dev;
> + int rc = 0;
> +
> + mutex_lock(&device_mutex);
> + list_for_each_entry(dev, &device_list, dev_list) {
> + if (dev->feature && dev->feature(dev)) {
What happens when features change? or when the device goes down?
> + ctx = create_ctx(sk);
> + if (!ctx)
> + goto out;
> +
> + ctx->hash = sk->sk_prot->hash;
> + ctx->unhash = sk->sk_prot->unhash;
> + ctx->sk_proto_close = sk->sk_prot->close;
> + ctx->conf = TLS_HW_RECORD;
> + update_sk_prot(sk, ctx);
> + rc = 1;
> + break;
> + }
> + }
> +out:
> + mutex_unlock(&device_mutex);
> + return rc;
> +}
> +
> +static void tls_hw_unhash(struct sock *sk)
> +{
> + struct tls_context *ctx = tls_get_ctx(sk);
> + struct tls_device *dev;
> +
> + mutex_lock(&device_mutex);
> + list_for_each_entry(dev, &device_list, dev_list) {
> + if (dev->unhash)
> + dev->unhash(dev, sk);
> + }
> + mutex_unlock(&device_mutex);
> + ctx->unhash(sk);
> +}
> +
> +static int tls_hw_hash(struct sock *sk)
> +{
> + struct tls_context *ctx = tls_get_ctx(sk);
> + struct tls_device *dev;
> + int err;
> +
> + err = ctx->hash(sk);
> + mutex_lock(&device_mutex);
> + list_for_each_entry(dev, &device_list, dev_list) {
> + if (dev->hash)
> + err |= dev->hash(dev, sk);
> + }
> + mutex_unlock(&device_mutex);
> +
> + if (err)
> + tls_hw_unhash(sk);
> + return err;
> +}
> +
> static void build_protos(struct proto *prot, struct proto *base)
> {
> prot[TLS_BASE] = *base;
> @@ -511,15 +597,22 @@ static void build_protos(struct proto *prot, struct proto *base)
> prot[TLS_SW_RXTX] = prot[TLS_SW_TX];
> prot[TLS_SW_RXTX].recvmsg = tls_sw_recvmsg;
> prot[TLS_SW_RXTX].close = tls_sk_proto_close;
> +
> + prot[TLS_HW_RECORD] = *base;
> + prot[TLS_HW_RECORD].hash = tls_hw_hash;
> + prot[TLS_HW_RECORD].unhash = tls_hw_unhash;
> + prot[TLS_HW_RECORD].close = tls_sk_proto_close;
> }
>
> static int tls_init(struct sock *sk)
> {
> int ip_ver = sk->sk_family == AF_INET6 ? TLSV6 : TLSV4;
> - struct inet_connection_sock *icsk = inet_csk(sk);
> struct tls_context *ctx;
> int rc = 0;
>
> + if (tls_hw_prot(sk))
> + goto out;
> +
> /* The TLS ulp is currently supported only for TCP sockets
> * in ESTABLISHED state.
> * Supporting sockets in LISTEN state will require us
> @@ -530,12 +623,11 @@ static int tls_init(struct sock *sk)
> return -ENOTSUPP;
>
> /* allocate tls context */
> - ctx = kzalloc(sizeof(*ctx), GFP_KERNEL);
> + ctx = create_ctx(sk);
> if (!ctx) {
> rc = -ENOMEM;
> goto out;
> }
> - icsk->icsk_ulp_data = ctx;
> ctx->setsockopt = sk->sk_prot->setsockopt;
> ctx->getsockopt = sk->sk_prot->getsockopt;
> ctx->sk_proto_close = sk->sk_prot->close;
> @@ -557,6 +649,22 @@ static int tls_init(struct sock *sk)
> return rc;
> }
>
> +void tls_register_device(struct tls_device *device)
> +{
> + mutex_lock(&device_mutex);
> + list_add_tail(&device->dev_list, &device_list);
> + mutex_unlock(&device_mutex);
> +}
> +EXPORT_SYMBOL(tls_register_device);
> +
> +void tls_unregister_device(struct tls_device *device)
> +{
> + mutex_lock(&device_mutex);
> + list_del(&device->dev_list);
> + mutex_unlock(&device_mutex);
> +}
> +EXPORT_SYMBOL(tls_unregister_device);
> +
> static struct tcp_ulp_ops tcp_tls_ulp_ops __read_mostly = {
> .name = "tls",
> .uid = TCP_ULP_TLS,
>
^ permalink raw reply
* [RFC v2] virtio: support packed ring
From: Tiwei Bie @ 2018-04-01 14:12 UTC (permalink / raw)
To: mst, jasowang, wexu, virtualization, linux-kernel, netdev
Cc: jfreimann, tiwei.bie
Hello everyone,
This RFC implements packed ring support for virtio driver.
The code was tested with DPDK vhost (testpmd/vhost-PMD) implemented
by Jens at http://dpdk.org/ml/archives/dev/2018-January/089417.html
Minor changes are needed for the vhost code, e.g. to kick the guest.
TODO:
- Refinements and bug fixes;
- Split into small patches;
- Test indirect descriptor support;
- Test/fix event suppression support;
- Test devices other than net;
RFC v1 -> RFC v2:
- Add indirect descriptor support - compile test only;
- Add event suppression supprt - compile test only;
- Move vring_packed_init() out of uapi (Jason, MST);
- Merge two loops into one in virtqueue_add_packed() (Jason);
- Split vring_unmap_one() for packed ring and split ring (Jason);
- Avoid using '%' operator (Jason);
- Rename free_head -> next_avail_idx (Jason);
- Add comments for virtio_wmb() in virtqueue_add_packed() (Jason);
- Some other refinements and bug fixes;
Thanks!
Signed-off-by: Tiwei Bie <tiwei.bie@intel.com>
---
drivers/virtio/virtio_ring.c | 1094 +++++++++++++++++++++++++++++-------
include/linux/virtio_ring.h | 8 +-
include/uapi/linux/virtio_config.h | 12 +-
include/uapi/linux/virtio_ring.h | 61 ++
4 files changed, 980 insertions(+), 195 deletions(-)
diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c
index 71458f493cf8..0515dca34d77 100644
--- a/drivers/virtio/virtio_ring.c
+++ b/drivers/virtio/virtio_ring.c
@@ -58,14 +58,15 @@
struct vring_desc_state {
void *data; /* Data for callback. */
- struct vring_desc *indir_desc; /* Indirect descriptor, if any. */
+ void *indir_desc; /* Indirect descriptor, if any. */
+ int num; /* Descriptor list length. */
};
struct vring_virtqueue {
struct virtqueue vq;
- /* Actual memory layout for this queue */
- struct vring vring;
+ /* Is this a packed ring? */
+ bool packed;
/* Can we use weak barriers? */
bool weak_barriers;
@@ -79,19 +80,45 @@ struct vring_virtqueue {
/* Host publishes avail event idx */
bool event;
- /* Head of free buffer list. */
- unsigned int free_head;
/* Number we've added since last sync. */
unsigned int num_added;
/* Last used index we've seen. */
u16 last_used_idx;
- /* Last written value to avail->flags */
- u16 avail_flags_shadow;
+ union {
+ /* Available for split ring */
+ struct {
+ /* Actual memory layout for this queue. */
+ struct vring vring;
- /* Last written value to avail->idx in guest byte order */
- u16 avail_idx_shadow;
+ /* Head of free buffer list. */
+ unsigned int free_head;
+
+ /* Last written value to avail->flags */
+ u16 avail_flags_shadow;
+
+ /* Last written value to avail->idx in
+ * guest byte order. */
+ u16 avail_idx_shadow;
+ };
+
+ /* Available for packed ring */
+ struct {
+ /* Actual memory layout for this queue. */
+ struct vring_packed vring_packed;
+
+ /* Driver ring wrap counter. */
+ u8 wrap_counter;
+
+ /* Index of the next avail descriptor. */
+ unsigned int next_avail_idx;
+
+ /* Last written value to driver->flags in
+ * guest byte order. */
+ u16 event_flags_shadow;
+ };
+ };
/* How to notify other side. FIXME: commonalize hcalls! */
bool (*notify)(struct virtqueue *vq);
@@ -201,8 +228,33 @@ static dma_addr_t vring_map_single(const struct vring_virtqueue *vq,
cpu_addr, size, direction);
}
-static void vring_unmap_one(const struct vring_virtqueue *vq,
- struct vring_desc *desc)
+static void vring_unmap_one_split(const struct vring_virtqueue *vq,
+ struct vring_desc *desc)
+{
+ u16 flags;
+
+ if (!vring_use_dma_api(vq->vq.vdev))
+ return;
+
+ flags = virtio16_to_cpu(vq->vq.vdev, desc->flags);
+
+ if (flags & VRING_DESC_F_INDIRECT) {
+ dma_unmap_single(vring_dma_dev(vq),
+ virtio64_to_cpu(vq->vq.vdev, desc->addr),
+ virtio32_to_cpu(vq->vq.vdev, desc->len),
+ (flags & VRING_DESC_F_WRITE) ?
+ DMA_FROM_DEVICE : DMA_TO_DEVICE);
+ } else {
+ dma_unmap_page(vring_dma_dev(vq),
+ virtio64_to_cpu(vq->vq.vdev, desc->addr),
+ virtio32_to_cpu(vq->vq.vdev, desc->len),
+ (flags & VRING_DESC_F_WRITE) ?
+ DMA_FROM_DEVICE : DMA_TO_DEVICE);
+ }
+}
+
+static void vring_unmap_one_packed(const struct vring_virtqueue *vq,
+ struct vring_packed_desc *desc)
{
u16 flags;
@@ -235,8 +287,9 @@ static int vring_mapping_error(const struct vring_virtqueue *vq,
return dma_mapping_error(vring_dma_dev(vq), addr);
}
-static struct vring_desc *alloc_indirect(struct virtqueue *_vq,
- unsigned int total_sg, gfp_t gfp)
+static struct vring_desc *alloc_indirect_split(struct virtqueue *_vq,
+ unsigned int total_sg,
+ gfp_t gfp)
{
struct vring_desc *desc;
unsigned int i;
@@ -257,14 +310,32 @@ static struct vring_desc *alloc_indirect(struct virtqueue *_vq,
return desc;
}
-static inline int virtqueue_add(struct virtqueue *_vq,
- struct scatterlist *sgs[],
- unsigned int total_sg,
- unsigned int out_sgs,
- unsigned int in_sgs,
- void *data,
- void *ctx,
- gfp_t gfp)
+static struct vring_packed_desc *alloc_indirect_packed(struct virtqueue *_vq,
+ unsigned int total_sg,
+ gfp_t gfp)
+{
+ struct vring_packed_desc *desc;
+
+ /*
+ * We require lowmem mappings for the descriptors because
+ * otherwise virt_to_phys will give us bogus addresses in the
+ * virtqueue.
+ */
+ gfp &= ~__GFP_HIGHMEM;
+
+ desc = kmalloc(total_sg * sizeof(struct vring_packed_desc), gfp);
+
+ return desc;
+}
+
+static inline int virtqueue_add_split(struct virtqueue *_vq,
+ struct scatterlist *sgs[],
+ unsigned int total_sg,
+ unsigned int out_sgs,
+ unsigned int in_sgs,
+ void *data,
+ void *ctx,
+ gfp_t gfp)
{
struct vring_virtqueue *vq = to_vvq(_vq);
struct scatterlist *sg;
@@ -303,7 +374,7 @@ static inline int virtqueue_add(struct virtqueue *_vq,
/* If the host supports indirect descriptor tables, and we have multiple
* buffers, then go indirect. FIXME: tune this threshold */
if (vq->indirect && total_sg > 1 && vq->vq.num_free)
- desc = alloc_indirect(_vq, total_sg, gfp);
+ desc = alloc_indirect_split(_vq, total_sg, gfp);
else {
desc = NULL;
WARN_ON_ONCE(total_sg > vq->vring.num && !vq->indirect);
@@ -424,7 +495,7 @@ static inline int virtqueue_add(struct virtqueue *_vq,
for (n = 0; n < total_sg; n++) {
if (i == err_idx)
break;
- vring_unmap_one(vq, &desc[i]);
+ vring_unmap_one_split(vq, &desc[i]);
i = virtio16_to_cpu(_vq->vdev, vq->vring.desc[i].next);
}
@@ -435,6 +506,210 @@ static inline int virtqueue_add(struct virtqueue *_vq,
return -EIO;
}
+static inline int virtqueue_add_packed(struct virtqueue *_vq,
+ struct scatterlist *sgs[],
+ unsigned int total_sg,
+ unsigned int out_sgs,
+ unsigned int in_sgs,
+ void *data,
+ void *ctx,
+ gfp_t gfp)
+{
+ struct vring_virtqueue *vq = to_vvq(_vq);
+ struct vring_packed_desc *desc;
+ struct scatterlist *sg;
+ unsigned int i, n, descs_used, uninitialized_var(prev), err_idx;
+ __virtio16 uninitialized_var(head_flags), flags;
+ int head, wrap_counter;
+ bool indirect;
+
+ START_USE(vq);
+
+ BUG_ON(data == NULL);
+ BUG_ON(ctx && vq->indirect);
+
+ if (unlikely(vq->broken)) {
+ END_USE(vq);
+ return -EIO;
+ }
+
+#ifdef DEBUG
+ {
+ ktime_t now = ktime_get();
+
+ /* No kick or get, with .1 second between? Warn. */
+ if (vq->last_add_time_valid)
+ WARN_ON(ktime_to_ms(ktime_sub(now, vq->last_add_time))
+ > 100);
+ vq->last_add_time = now;
+ vq->last_add_time_valid = true;
+ }
+#endif
+
+ BUG_ON(total_sg == 0);
+
+ head = vq->next_avail_idx;
+ wrap_counter = vq->wrap_counter;
+
+ /* If the host supports indirect descriptor tables, and we have multiple
+ * buffers, then go indirect. FIXME: tune this threshold */
+ if (vq->indirect && total_sg > 1 && vq->vq.num_free)
+ desc = alloc_indirect_packed(_vq, total_sg, gfp);
+ else {
+ desc = NULL;
+ WARN_ON_ONCE(total_sg > vq->vring_packed.num && !vq->indirect);
+ }
+
+ if (desc) {
+ /* Use a single buffer which doesn't continue */
+ indirect = true;
+ /* Set up rest to use this indirect table. */
+ i = 0;
+ descs_used = 1;
+ } else {
+ indirect = false;
+ desc = vq->vring_packed.desc;
+ i = head;
+ descs_used = total_sg;
+ }
+
+ if (vq->vq.num_free < descs_used) {
+ pr_debug("Can't add buf len %i - avail = %i\n",
+ descs_used, vq->vq.num_free);
+ /* FIXME: for historical reasons, we force a notify here if
+ * there are outgoing parts to the buffer. Presumably the
+ * host should service the ring ASAP. */
+ if (out_sgs)
+ vq->notify(&vq->vq);
+ if (indirect)
+ kfree(desc);
+ END_USE(vq);
+ return -ENOSPC;
+ }
+
+ for (n = 0; n < out_sgs + in_sgs; n++) {
+ for (sg = sgs[n]; sg; sg = sg_next(sg)) {
+ dma_addr_t addr = vring_map_one_sg(vq, sg, n < out_sgs ?
+ DMA_TO_DEVICE : DMA_FROM_DEVICE);
+ if (vring_mapping_error(vq, addr))
+ goto unmap_release;
+
+ flags = cpu_to_virtio16(_vq->vdev, VRING_DESC_F_NEXT |
+ (n < out_sgs ? 0 : VRING_DESC_F_WRITE) |
+ VRING_DESC_F_AVAIL(vq->wrap_counter) |
+ VRING_DESC_F_USED(!vq->wrap_counter));
+ if (!indirect && i == head)
+ head_flags = flags;
+ else
+ desc[i].flags = flags;
+
+ desc[i].addr = cpu_to_virtio64(_vq->vdev, addr);
+ desc[i].len = cpu_to_virtio32(_vq->vdev, sg->length);
+ desc[i].id = cpu_to_virtio32(_vq->vdev, head);
+ prev = i;
+ i++;
+ if (!indirect && i >= vq->vring_packed.num) {
+ i = 0;
+ vq->wrap_counter ^= 1;
+ }
+ }
+ }
+ /* Last one doesn't continue. */
+ if (total_sg == 1)
+ head_flags &= cpu_to_virtio16(_vq->vdev, ~VRING_DESC_F_NEXT);
+ else
+ desc[prev].flags &= cpu_to_virtio16(_vq->vdev, ~VRING_DESC_F_NEXT);
+
+ if (indirect) {
+ /* Now that the indirect table is filled in, map it. */
+ dma_addr_t addr = vring_map_single(
+ vq, desc, total_sg * sizeof(struct vring_packed_desc),
+ DMA_TO_DEVICE);
+ if (vring_mapping_error(vq, addr))
+ goto unmap_release;
+
+ head_flags = cpu_to_virtio16(_vq->vdev, VRING_DESC_F_INDIRECT |
+ VRING_DESC_F_AVAIL(wrap_counter) |
+ VRING_DESC_F_USED(!wrap_counter));
+ vq->vring_packed.desc[head].addr = cpu_to_virtio64(_vq->vdev, addr);
+ vq->vring_packed.desc[head].len = cpu_to_virtio32(_vq->vdev,
+ total_sg * sizeof(struct vring_packed_desc));
+ vq->vring_packed.desc[head].id = cpu_to_virtio32(_vq->vdev, head);
+ }
+
+ /* We're using some buffers from the free list. */
+ vq->vq.num_free -= descs_used;
+
+ /* Update free pointer */
+ if (indirect) {
+ n = head + 1;
+ if (n >= vq->vring_packed.num) {
+ n = 0;
+ vq->wrap_counter ^= 1;
+ }
+ vq->next_avail_idx = n;
+ } else
+ vq->next_avail_idx = i;
+
+ /* Store token and indirect buffer state. */
+ vq->desc_state[head].num = descs_used;
+ vq->desc_state[head].data = data;
+ if (indirect)
+ vq->desc_state[head].indir_desc = desc;
+ else
+ vq->desc_state[head].indir_desc = ctx;
+
+ /* A driver MUST NOT make the first descriptor in the list
+ * available before all subsequent descriptors comprising
+ * the list are made available. */
+ virtio_wmb(vq->weak_barriers);
+ vq->vring_packed.desc[head].flags = head_flags;
+ vq->num_added++;
+
+ pr_debug("Added buffer head %i to %p\n", head, vq);
+ END_USE(vq);
+
+ return 0;
+
+unmap_release:
+ err_idx = i;
+ i = head;
+
+ for (n = 0; n < total_sg; n++) {
+ if (i == err_idx)
+ break;
+ vring_unmap_one_packed(vq, &desc[i]);
+ i++;
+ if (!indirect && i >= vq->vring_packed.num)
+ i = 0;
+ }
+
+ vq->wrap_counter = wrap_counter;
+
+ if (indirect)
+ kfree(desc);
+
+ END_USE(vq);
+ return -EIO;
+}
+
+static inline int virtqueue_add(struct virtqueue *_vq,
+ struct scatterlist *sgs[],
+ unsigned int total_sg,
+ unsigned int out_sgs,
+ unsigned int in_sgs,
+ void *data,
+ void *ctx,
+ gfp_t gfp)
+{
+ struct vring_virtqueue *vq = to_vvq(_vq);
+
+ return vq->packed ? virtqueue_add_packed(_vq, sgs, total_sg, out_sgs,
+ in_sgs, data, ctx, gfp) :
+ virtqueue_add_split(_vq, sgs, total_sg, out_sgs,
+ in_sgs, data, ctx, gfp);
+}
+
/**
* virtqueue_add_sgs - expose buffers to other end
* @vq: the struct virtqueue we're talking about.
@@ -537,18 +812,7 @@ int virtqueue_add_inbuf_ctx(struct virtqueue *vq,
}
EXPORT_SYMBOL_GPL(virtqueue_add_inbuf_ctx);
-/**
- * virtqueue_kick_prepare - first half of split virtqueue_kick call.
- * @vq: the struct virtqueue
- *
- * Instead of virtqueue_kick(), you can do:
- * if (virtqueue_kick_prepare(vq))
- * virtqueue_notify(vq);
- *
- * This is sometimes useful because the virtqueue_kick_prepare() needs
- * to be serialized, but the actual virtqueue_notify() call does not.
- */
-bool virtqueue_kick_prepare(struct virtqueue *_vq)
+static bool virtqueue_kick_prepare_split(struct virtqueue *_vq)
{
struct vring_virtqueue *vq = to_vvq(_vq);
u16 new, old;
@@ -580,6 +844,62 @@ bool virtqueue_kick_prepare(struct virtqueue *_vq)
END_USE(vq);
return needs_kick;
}
+
+static bool virtqueue_kick_prepare_packed(struct virtqueue *_vq)
+{
+ struct vring_virtqueue *vq = to_vvq(_vq);
+ u16 new, old, off_wrap;
+ bool needs_kick;
+
+ START_USE(vq);
+ /* We need to expose the new flags value before checking notification
+ * suppressions. */
+ virtio_mb(vq->weak_barriers);
+
+ old = vq->next_avail_idx - vq->num_added;
+ new = vq->next_avail_idx;
+ vq->num_added = 0;
+
+#ifdef DEBUG
+ if (vq->last_add_time_valid) {
+ WARN_ON(ktime_to_ms(ktime_sub(ktime_get(),
+ vq->last_add_time)) > 100);
+ }
+ vq->last_add_time_valid = false;
+#endif
+
+ off_wrap = virtio16_to_cpu(_vq->vdev, vq->vring_packed.device->off_wrap);
+
+ if (vq->event) {
+ // FIXME: fix this!
+ needs_kick = ((off_wrap >> 15) == vq->wrap_counter) &&
+ vring_need_event(off_wrap & ~(1<<15), new, old);
+ } else {
+ needs_kick = (vq->vring_packed.device->flags !=
+ cpu_to_virtio16(_vq->vdev, VRING_EVENT_F_DISABLE));
+ }
+ END_USE(vq);
+ return needs_kick;
+}
+
+/**
+ * virtqueue_kick_prepare - first half of split virtqueue_kick call.
+ * @vq: the struct virtqueue
+ *
+ * Instead of virtqueue_kick(), you can do:
+ * if (virtqueue_kick_prepare(vq))
+ * virtqueue_notify(vq);
+ *
+ * This is sometimes useful because the virtqueue_kick_prepare() needs
+ * to be serialized, but the actual virtqueue_notify() call does not.
+ */
+bool virtqueue_kick_prepare(struct virtqueue *_vq)
+{
+ struct vring_virtqueue *vq = to_vvq(_vq);
+
+ return vq->packed ? virtqueue_kick_prepare_packed(_vq) :
+ virtqueue_kick_prepare_split(_vq);
+}
EXPORT_SYMBOL_GPL(virtqueue_kick_prepare);
/**
@@ -626,8 +946,8 @@ bool virtqueue_kick(struct virtqueue *vq)
}
EXPORT_SYMBOL_GPL(virtqueue_kick);
-static void detach_buf(struct vring_virtqueue *vq, unsigned int head,
- void **ctx)
+static void detach_buf_split(struct vring_virtqueue *vq, unsigned int head,
+ void **ctx)
{
unsigned int i, j;
__virtio16 nextflag = cpu_to_virtio16(vq->vq.vdev, VRING_DESC_F_NEXT);
@@ -639,12 +959,12 @@ static void detach_buf(struct vring_virtqueue *vq, unsigned int head,
i = head;
while (vq->vring.desc[i].flags & nextflag) {
- vring_unmap_one(vq, &vq->vring.desc[i]);
+ vring_unmap_one_split(vq, &vq->vring.desc[i]);
i = virtio16_to_cpu(vq->vq.vdev, vq->vring.desc[i].next);
vq->vq.num_free++;
}
- vring_unmap_one(vq, &vq->vring.desc[i]);
+ vring_unmap_one_split(vq, &vq->vring.desc[i]);
vq->vring.desc[i].next = cpu_to_virtio16(vq->vq.vdev, vq->free_head);
vq->free_head = head;
@@ -666,7 +986,7 @@ static void detach_buf(struct vring_virtqueue *vq, unsigned int head,
BUG_ON(len == 0 || len % sizeof(struct vring_desc));
for (j = 0; j < len / sizeof(struct vring_desc); j++)
- vring_unmap_one(vq, &indir_desc[j]);
+ vring_unmap_one_split(vq, &indir_desc[j]);
kfree(indir_desc);
vq->desc_state[head].indir_desc = NULL;
@@ -675,11 +995,207 @@ static void detach_buf(struct vring_virtqueue *vq, unsigned int head,
}
}
-static inline bool more_used(const struct vring_virtqueue *vq)
+static int detach_buf_packed(struct vring_virtqueue *vq, unsigned int head,
+ void **ctx)
+{
+ struct vring_packed_desc *desc;
+ unsigned int i, j;
+
+ /* Clear data ptr. */
+ vq->desc_state[head].data = NULL;
+
+ i = head;
+
+ for (j = 0; j < vq->desc_state[head].num; j++) {
+ desc = &vq->vring_packed.desc[i];
+ vring_unmap_one_packed(vq, desc);
+ desc->flags = 0x0;
+ i++;
+ if (i >= vq->vring_packed.num)
+ i = 0;
+ }
+
+ vq->vq.num_free += vq->desc_state[head].num;
+
+ if (vq->indirect) {
+ u32 len;
+
+ desc = vq->desc_state[head].indir_desc;
+ /* Free the indirect table, if any, now that it's unmapped. */
+ if (!desc)
+ goto out;
+
+ len = virtio32_to_cpu(vq->vq.vdev,
+ vq->vring_packed.desc[head].len);
+
+ BUG_ON(!(vq->vring_packed.desc[head].flags &
+ cpu_to_virtio16(vq->vq.vdev, VRING_DESC_F_INDIRECT)));
+ BUG_ON(len == 0 || len % sizeof(struct vring_packed_desc));
+
+ for (j = 0; j < len / sizeof(struct vring_packed_desc); j++)
+ vring_unmap_one_packed(vq, &desc[j]);
+
+ kfree(desc);
+ vq->desc_state[head].indir_desc = NULL;
+ } else if (ctx) {
+ *ctx = vq->desc_state[head].indir_desc;
+ }
+
+out:
+ return vq->desc_state[head].num;
+}
+
+static inline bool more_used_split(const struct vring_virtqueue *vq)
{
return vq->last_used_idx != virtio16_to_cpu(vq->vq.vdev, vq->vring.used->idx);
}
+static inline bool more_used_packed(const struct vring_virtqueue *vq)
+{
+ u16 last_used, flags;
+ bool avail, used;
+
+ if (vq->vq.num_free == vq->vring_packed.num)
+ return false;
+
+ last_used = vq->last_used_idx;
+ flags = virtio16_to_cpu(vq->vq.vdev,
+ vq->vring_packed.desc[last_used].flags);
+ avail = flags & VRING_DESC_F_AVAIL(1);
+ used = flags & VRING_DESC_F_USED(1);
+
+ return avail == used;
+}
+
+static inline bool more_used(const struct vring_virtqueue *vq)
+{
+ return vq->packed ? more_used_packed(vq) : more_used_split(vq);
+}
+
+void *virtqueue_get_buf_ctx_split(struct virtqueue *_vq, unsigned int *len,
+ void **ctx)
+{
+ struct vring_virtqueue *vq = to_vvq(_vq);
+ void *ret;
+ unsigned int i;
+ u16 last_used;
+
+ START_USE(vq);
+
+ if (unlikely(vq->broken)) {
+ END_USE(vq);
+ return NULL;
+ }
+
+ if (!more_used(vq)) {
+ pr_debug("No more buffers in queue\n");
+ END_USE(vq);
+ return NULL;
+ }
+
+ /* Only get used array entries after they have been exposed by host. */
+ virtio_rmb(vq->weak_barriers);
+
+ last_used = (vq->last_used_idx & (vq->vring.num - 1));
+ i = virtio32_to_cpu(_vq->vdev, vq->vring.used->ring[last_used].id);
+ *len = virtio32_to_cpu(_vq->vdev, vq->vring.used->ring[last_used].len);
+
+ if (unlikely(i >= vq->vring.num)) {
+ BAD_RING(vq, "id %u out of range\n", i);
+ return NULL;
+ }
+ if (unlikely(!vq->desc_state[i].data)) {
+ BAD_RING(vq, "id %u is not a head!\n", i);
+ return NULL;
+ }
+
+ /* detach_buf_split clears data, so grab it now. */
+ ret = vq->desc_state[i].data;
+ detach_buf_split(vq, i, ctx);
+ vq->last_used_idx++;
+ /* If we expect an interrupt for the next entry, tell host
+ * by writing event index and flush out the write before
+ * the read in the next get_buf call. */
+ if (!(vq->avail_flags_shadow & VRING_AVAIL_F_NO_INTERRUPT))
+ virtio_store_mb(vq->weak_barriers,
+ &vring_used_event(&vq->vring),
+ cpu_to_virtio16(_vq->vdev, vq->last_used_idx));
+
+#ifdef DEBUG
+ vq->last_add_time_valid = false;
+#endif
+
+ END_USE(vq);
+ return ret;
+}
+
+void *virtqueue_get_buf_ctx_packed(struct virtqueue *_vq, unsigned int *len,
+ void **ctx)
+{
+ struct vring_virtqueue *vq = to_vvq(_vq);
+ uint16_t wrap_counter;
+ void *ret;
+ unsigned int i;
+ u16 last_used;
+
+ START_USE(vq);
+
+ if (unlikely(vq->broken)) {
+ END_USE(vq);
+ return NULL;
+ }
+
+ if (!more_used(vq)) {
+ pr_debug("No more buffers in queue\n");
+ END_USE(vq);
+ return NULL;
+ }
+
+ /* Only get used elements after they have been exposed by host. */
+ virtio_rmb(vq->weak_barriers);
+
+ last_used = vq->last_used_idx;
+ i = virtio32_to_cpu(_vq->vdev, vq->vring_packed.desc[last_used].id);
+ *len = virtio32_to_cpu(_vq->vdev, vq->vring_packed.desc[last_used].len);
+
+ if (unlikely(i >= vq->vring_packed.num)) {
+ BAD_RING(vq, "id %u out of range\n", i);
+ return NULL;
+ }
+ if (unlikely(!vq->desc_state[i].data)) {
+ BAD_RING(vq, "id %u is not a head!\n", i);
+ return NULL;
+ }
+
+ /* detach_buf_packed clears data, so grab it now. */
+ ret = vq->desc_state[i].data;
+ detach_buf_packed(vq, i, ctx);
+
+ vq->last_used_idx += vq->desc_state[i].num;
+ if (vq->last_used_idx >= vq->vring_packed.num)
+ vq->last_used_idx -= vq->vring_packed.num;
+
+ wrap_counter = vq->wrap_counter;
+ if (vq->last_used_idx > vq->next_avail_idx)
+ wrap_counter ^= 1;
+
+ /* If we expect an interrupt for the next entry, tell host
+ * by writing event index and flush out the write before
+ * the read in the next get_buf call. */
+ if (vq->event_flags_shadow == VRING_EVENT_F_DESC)
+ virtio_store_mb(vq->weak_barriers,
+ &vq->vring_packed.driver->off_wrap,
+ cpu_to_virtio16(_vq->vdev, vq->last_used_idx |
+ wrap_counter << 15));
+
+#ifdef DEBUG
+ vq->last_add_time_valid = false;
+#endif
+
+ END_USE(vq);
+ return ret;
+}
+
/**
* virtqueue_get_buf - get the next used buffer
* @vq: the struct virtqueue we're talking about.
@@ -700,57 +1216,9 @@ void *virtqueue_get_buf_ctx(struct virtqueue *_vq, unsigned int *len,
void **ctx)
{
struct vring_virtqueue *vq = to_vvq(_vq);
- void *ret;
- unsigned int i;
- u16 last_used;
- START_USE(vq);
-
- if (unlikely(vq->broken)) {
- END_USE(vq);
- return NULL;
- }
-
- if (!more_used(vq)) {
- pr_debug("No more buffers in queue\n");
- END_USE(vq);
- return NULL;
- }
-
- /* Only get used array entries after they have been exposed by host. */
- virtio_rmb(vq->weak_barriers);
-
- last_used = (vq->last_used_idx & (vq->vring.num - 1));
- i = virtio32_to_cpu(_vq->vdev, vq->vring.used->ring[last_used].id);
- *len = virtio32_to_cpu(_vq->vdev, vq->vring.used->ring[last_used].len);
-
- if (unlikely(i >= vq->vring.num)) {
- BAD_RING(vq, "id %u out of range\n", i);
- return NULL;
- }
- if (unlikely(!vq->desc_state[i].data)) {
- BAD_RING(vq, "id %u is not a head!\n", i);
- return NULL;
- }
-
- /* detach_buf clears data, so grab it now. */
- ret = vq->desc_state[i].data;
- detach_buf(vq, i, ctx);
- vq->last_used_idx++;
- /* If we expect an interrupt for the next entry, tell host
- * by writing event index and flush out the write before
- * the read in the next get_buf call. */
- if (!(vq->avail_flags_shadow & VRING_AVAIL_F_NO_INTERRUPT))
- virtio_store_mb(vq->weak_barriers,
- &vring_used_event(&vq->vring),
- cpu_to_virtio16(_vq->vdev, vq->last_used_idx));
-
-#ifdef DEBUG
- vq->last_add_time_valid = false;
-#endif
-
- END_USE(vq);
- return ret;
+ return vq->packed ? virtqueue_get_buf_ctx_packed(_vq, len, ctx) :
+ virtqueue_get_buf_ctx_split(_vq, len, ctx);
}
EXPORT_SYMBOL_GPL(virtqueue_get_buf_ctx);
@@ -759,6 +1227,29 @@ void *virtqueue_get_buf(struct virtqueue *_vq, unsigned int *len)
return virtqueue_get_buf_ctx(_vq, len, NULL);
}
EXPORT_SYMBOL_GPL(virtqueue_get_buf);
+
+static void virtqueue_disable_cb_split(struct virtqueue *_vq)
+{
+ struct vring_virtqueue *vq = to_vvq(_vq);
+
+ if (!(vq->avail_flags_shadow & VRING_AVAIL_F_NO_INTERRUPT)) {
+ vq->avail_flags_shadow |= VRING_AVAIL_F_NO_INTERRUPT;
+ if (!vq->event)
+ vq->vring.avail->flags = cpu_to_virtio16(_vq->vdev, vq->avail_flags_shadow);
+ }
+}
+
+static void virtqueue_disable_cb_packed(struct virtqueue *_vq)
+{
+ struct vring_virtqueue *vq = to_vvq(_vq);
+
+ if (vq->event_flags_shadow != VRING_EVENT_F_DISABLE) {
+ vq->event_flags_shadow = VRING_EVENT_F_DISABLE;
+ vq->vring_packed.driver->flags = cpu_to_virtio16(_vq->vdev,
+ vq->event_flags_shadow);
+ }
+}
+
/**
* virtqueue_disable_cb - disable callbacks
* @vq: the struct virtqueue we're talking about.
@@ -772,15 +1263,66 @@ void virtqueue_disable_cb(struct virtqueue *_vq)
{
struct vring_virtqueue *vq = to_vvq(_vq);
- if (!(vq->avail_flags_shadow & VRING_AVAIL_F_NO_INTERRUPT)) {
- vq->avail_flags_shadow |= VRING_AVAIL_F_NO_INTERRUPT;
- if (!vq->event)
- vq->vring.avail->flags = cpu_to_virtio16(_vq->vdev, vq->avail_flags_shadow);
- }
-
+ if (vq->packed)
+ virtqueue_disable_cb_packed(_vq);
+ else
+ virtqueue_disable_cb_split(_vq);
}
EXPORT_SYMBOL_GPL(virtqueue_disable_cb);
+static unsigned virtqueue_enable_cb_prepare_split(struct virtqueue *_vq)
+{
+ struct vring_virtqueue *vq = to_vvq(_vq);
+ u16 last_used_idx;
+
+ START_USE(vq);
+
+ /* We optimistically turn back on interrupts, then check if there was
+ * more to do. */
+ /* Depending on the VIRTIO_RING_F_EVENT_IDX feature, we need to
+ * either clear the flags bit or point the event index at the next
+ * entry. Always do both to keep code simple. */
+ if (vq->avail_flags_shadow & VRING_AVAIL_F_NO_INTERRUPT) {
+ vq->avail_flags_shadow &= ~VRING_AVAIL_F_NO_INTERRUPT;
+ if (!vq->event)
+ vq->vring.avail->flags = cpu_to_virtio16(_vq->vdev, vq->avail_flags_shadow);
+ }
+ vring_used_event(&vq->vring) = cpu_to_virtio16(_vq->vdev, last_used_idx = vq->last_used_idx);
+ END_USE(vq);
+ return last_used_idx;
+}
+
+static unsigned virtqueue_enable_cb_prepare_packed(struct virtqueue *_vq)
+{
+ struct vring_virtqueue *vq = to_vvq(_vq);
+ u16 last_used_idx, wrap_counter, off_wrap;
+
+ START_USE(vq);
+
+ last_used_idx = vq->last_used_idx;
+ wrap_counter = vq->wrap_counter;
+
+ if (last_used_idx > vq->next_avail_idx)
+ wrap_counter ^= 1;
+
+ off_wrap = last_used_idx | (wrap_counter << 15);
+
+ /* We optimistically turn back on interrupts, then check if there was
+ * more to do. */
+ /* Depending on the VIRTIO_RING_F_EVENT_IDX feature, we need to
+ * either clear the flags bit or point the event index at the next
+ * entry. Always do both to keep code simple. */
+ if (vq->event_flags_shadow == VRING_EVENT_F_DISABLE) {
+ vq->event_flags_shadow = vq->event ? VRING_EVENT_F_DESC:
+ VRING_EVENT_F_ENABLE;
+ vq->vring_packed.driver->flags = cpu_to_virtio16(_vq->vdev,
+ vq->event_flags_shadow);
+ }
+ vq->vring_packed.driver->off_wrap = cpu_to_virtio16(_vq->vdev, off_wrap);
+ END_USE(vq);
+ return last_used_idx;
+}
+
/**
* virtqueue_enable_cb_prepare - restart callbacks after disable_cb
* @vq: the struct virtqueue we're talking about.
@@ -796,26 +1338,34 @@ EXPORT_SYMBOL_GPL(virtqueue_disable_cb);
unsigned virtqueue_enable_cb_prepare(struct virtqueue *_vq)
{
struct vring_virtqueue *vq = to_vvq(_vq);
- u16 last_used_idx;
- START_USE(vq);
-
- /* We optimistically turn back on interrupts, then check if there was
- * more to do. */
- /* Depending on the VIRTIO_RING_F_EVENT_IDX feature, we need to
- * either clear the flags bit or point the event index at the next
- * entry. Always do both to keep code simple. */
- if (vq->avail_flags_shadow & VRING_AVAIL_F_NO_INTERRUPT) {
- vq->avail_flags_shadow &= ~VRING_AVAIL_F_NO_INTERRUPT;
- if (!vq->event)
- vq->vring.avail->flags = cpu_to_virtio16(_vq->vdev, vq->avail_flags_shadow);
- }
- vring_used_event(&vq->vring) = cpu_to_virtio16(_vq->vdev, last_used_idx = vq->last_used_idx);
- END_USE(vq);
- return last_used_idx;
+ return vq->packed ? virtqueue_enable_cb_prepare_packed(_vq) :
+ virtqueue_enable_cb_prepare_split(_vq);
}
EXPORT_SYMBOL_GPL(virtqueue_enable_cb_prepare);
+static bool virtqueue_poll_split(struct virtqueue *_vq, unsigned last_used_idx)
+{
+ struct vring_virtqueue *vq = to_vvq(_vq);
+
+ virtio_mb(vq->weak_barriers);
+ return (u16)last_used_idx != virtio16_to_cpu(_vq->vdev, vq->vring.used->idx);
+}
+
+static bool virtqueue_poll_packed(struct virtqueue *_vq, unsigned last_used_idx)
+{
+ struct vring_virtqueue *vq = to_vvq(_vq);
+ bool avail, used;
+ u16 flags;
+
+ virtio_mb(vq->weak_barriers);
+ flags = virtio16_to_cpu(vq->vq.vdev,
+ vq->vring_packed.desc[last_used_idx].flags);
+ avail = flags & VRING_DESC_F_AVAIL(1);
+ used = flags & VRING_DESC_F_USED(1);
+ return avail == used;
+}
+
/**
* virtqueue_poll - query pending used buffers
* @vq: the struct virtqueue we're talking about.
@@ -829,8 +1379,8 @@ bool virtqueue_poll(struct virtqueue *_vq, unsigned last_used_idx)
{
struct vring_virtqueue *vq = to_vvq(_vq);
- virtio_mb(vq->weak_barriers);
- return (u16)last_used_idx != virtio16_to_cpu(_vq->vdev, vq->vring.used->idx);
+ return vq->packed ? virtqueue_poll_packed(_vq, last_used_idx) :
+ virtqueue_poll_split(_vq, last_used_idx);
}
EXPORT_SYMBOL_GPL(virtqueue_poll);
@@ -852,6 +1402,83 @@ bool virtqueue_enable_cb(struct virtqueue *_vq)
}
EXPORT_SYMBOL_GPL(virtqueue_enable_cb);
+static bool virtqueue_enable_cb_delayed_split(struct virtqueue *_vq)
+{
+ struct vring_virtqueue *vq = to_vvq(_vq);
+ u16 bufs;
+
+ START_USE(vq);
+
+ /* We optimistically turn back on interrupts, then check if there was
+ * more to do. */
+ /* Depending on the VIRTIO_RING_F_USED_EVENT_IDX feature, we need to
+ * either clear the flags bit or point the event index at the next
+ * entry. Always update the event index to keep code simple. */
+ if (vq->avail_flags_shadow & VRING_AVAIL_F_NO_INTERRUPT) {
+ vq->avail_flags_shadow &= ~VRING_AVAIL_F_NO_INTERRUPT;
+ if (!vq->event)
+ vq->vring.avail->flags = cpu_to_virtio16(_vq->vdev, vq->avail_flags_shadow);
+ }
+ /* TODO: tune this threshold */
+ bufs = (u16)(vq->avail_idx_shadow - vq->last_used_idx) * 3 / 4;
+
+ virtio_store_mb(vq->weak_barriers,
+ &vring_used_event(&vq->vring),
+ cpu_to_virtio16(_vq->vdev, vq->last_used_idx + bufs));
+
+ if (unlikely((u16)(virtio16_to_cpu(_vq->vdev, vq->vring.used->idx) - vq->last_used_idx) > bufs)) {
+ END_USE(vq);
+ return false;
+ }
+
+ END_USE(vq);
+ return true;
+}
+
+static bool virtqueue_enable_cb_delayed_packed(struct virtqueue *_vq)
+{
+ struct vring_virtqueue *vq = to_vvq(_vq);
+ u16 bufs, off_wrap, used_idx, wrap_counter;
+
+ START_USE(vq);
+
+ /* We optimistically turn back on interrupts, then check if there was
+ * more to do. */
+ /* Depending on the VIRTIO_RING_F_USED_EVENT_IDX feature, we need to
+ * either clear the flags bit or point the event index at the next
+ * entry. Always update the event index to keep code simple. */
+ if (vq->event_flags_shadow == VRING_EVENT_F_DISABLE) {
+ vq->event_flags_shadow = vq->event ? VRING_EVENT_F_DESC:
+ VRING_EVENT_F_ENABLE;
+ vq->vring_packed.driver->flags = cpu_to_virtio16(_vq->vdev,
+ vq->event_flags_shadow);
+ }
+
+ /* TODO: tune this threshold */
+ bufs = (u16)(vq->next_avail_idx - vq->last_used_idx) * 3 / 4;
+
+ used_idx = vq->last_used_idx + bufs;
+ if (used_idx >= vq->vring_packed.num)
+ used_idx -= vq->vring_packed.num;
+
+ wrap_counter = vq->wrap_counter;
+ if (used_idx > vq->next_avail_idx)
+ wrap_counter ^= 1;
+
+ off_wrap = used_idx | (wrap_counter << 15);
+
+ virtio_store_mb(vq->weak_barriers, &vq->vring_packed.driver->off_wrap,
+ cpu_to_virtio16(_vq->vdev, off_wrap));
+
+ if (more_used_packed(vq)) {
+ END_USE(vq);
+ return false;
+ }
+
+ END_USE(vq);
+ return true;
+}
+
/**
* virtqueue_enable_cb_delayed - restart callbacks after disable_cb.
* @vq: the struct virtqueue we're talking about.
@@ -868,37 +1495,69 @@ EXPORT_SYMBOL_GPL(virtqueue_enable_cb);
bool virtqueue_enable_cb_delayed(struct virtqueue *_vq)
{
struct vring_virtqueue *vq = to_vvq(_vq);
- u16 bufs;
- START_USE(vq);
-
- /* We optimistically turn back on interrupts, then check if there was
- * more to do. */
- /* Depending on the VIRTIO_RING_F_USED_EVENT_IDX feature, we need to
- * either clear the flags bit or point the event index at the next
- * entry. Always update the event index to keep code simple. */
- if (vq->avail_flags_shadow & VRING_AVAIL_F_NO_INTERRUPT) {
- vq->avail_flags_shadow &= ~VRING_AVAIL_F_NO_INTERRUPT;
- if (!vq->event)
- vq->vring.avail->flags = cpu_to_virtio16(_vq->vdev, vq->avail_flags_shadow);
- }
- /* TODO: tune this threshold */
- bufs = (u16)(vq->avail_idx_shadow - vq->last_used_idx) * 3 / 4;
-
- virtio_store_mb(vq->weak_barriers,
- &vring_used_event(&vq->vring),
- cpu_to_virtio16(_vq->vdev, vq->last_used_idx + bufs));
-
- if (unlikely((u16)(virtio16_to_cpu(_vq->vdev, vq->vring.used->idx) - vq->last_used_idx) > bufs)) {
- END_USE(vq);
- return false;
- }
-
- END_USE(vq);
- return true;
+ return vq->packed ? virtqueue_enable_cb_delayed_packed(_vq) :
+ virtqueue_enable_cb_delayed_split(_vq);
}
EXPORT_SYMBOL_GPL(virtqueue_enable_cb_delayed);
+static void *virtqueue_detach_unused_buf_split(struct virtqueue *_vq)
+{
+ struct vring_virtqueue *vq = to_vvq(_vq);
+ unsigned int i;
+ void *buf;
+
+ START_USE(vq);
+
+ for (i = 0; i < vq->vring.num; i++) {
+ if (!vq->desc_state[i].data)
+ continue;
+ /* detach_buf clears data, so grab it now. */
+ buf = vq->desc_state[i].data;
+ detach_buf_split(vq, i, NULL);
+ vq->avail_idx_shadow--;
+ vq->vring.avail->idx = cpu_to_virtio16(_vq->vdev, vq->avail_idx_shadow);
+ END_USE(vq);
+ return buf;
+ }
+ /* That should have freed everything. */
+ BUG_ON(vq->vq.num_free != vq->vring.num);
+
+ END_USE(vq);
+ return NULL;
+}
+
+static void *virtqueue_detach_unused_buf_packed(struct virtqueue *_vq)
+{
+ struct vring_virtqueue *vq = to_vvq(_vq);
+ unsigned int i, num;
+ void *buf;
+
+ START_USE(vq);
+
+ for (i = 0; i < vq->vring_packed.num; i++) {
+ if (!vq->desc_state[i].data)
+ continue;
+ /* detach_buf clears data, so grab it now. */
+ buf = vq->desc_state[i].data;
+ num = detach_buf_packed(vq, i, NULL);
+ if (vq->next_avail_idx < num) {
+ vq->next_avail_idx = vq->vring_packed.num -
+ (num - vq->next_avail_idx);
+ vq->wrap_counter ^= 1;
+ } else {
+ vq->next_avail_idx -= num;
+ }
+ END_USE(vq);
+ return buf;
+ }
+ /* That should have freed everything. */
+ BUG_ON(vq->vq.num_free != vq->vring_packed.num);
+
+ END_USE(vq);
+ return NULL;
+}
+
/**
* virtqueue_detach_unused_buf - detach first unused buffer
* @vq: the struct virtqueue we're talking about.
@@ -910,27 +1569,9 @@ EXPORT_SYMBOL_GPL(virtqueue_enable_cb_delayed);
void *virtqueue_detach_unused_buf(struct virtqueue *_vq)
{
struct vring_virtqueue *vq = to_vvq(_vq);
- unsigned int i;
- void *buf;
- START_USE(vq);
-
- for (i = 0; i < vq->vring.num; i++) {
- if (!vq->desc_state[i].data)
- continue;
- /* detach_buf clears data, so grab it now. */
- buf = vq->desc_state[i].data;
- detach_buf(vq, i, NULL);
- vq->avail_idx_shadow--;
- vq->vring.avail->idx = cpu_to_virtio16(_vq->vdev, vq->avail_idx_shadow);
- END_USE(vq);
- return buf;
- }
- /* That should have freed everything. */
- BUG_ON(vq->vq.num_free != vq->vring.num);
-
- END_USE(vq);
- return NULL;
+ return vq->packed ? virtqueue_detach_unused_buf_packed(_vq) :
+ virtqueue_detach_unused_buf_split(_vq);
}
EXPORT_SYMBOL_GPL(virtqueue_detach_unused_buf);
@@ -955,7 +1596,8 @@ irqreturn_t vring_interrupt(int irq, void *_vq)
EXPORT_SYMBOL_GPL(vring_interrupt);
struct virtqueue *__vring_new_virtqueue(unsigned int index,
- struct vring vring,
+ union vring_union vring,
+ bool packed,
struct virtio_device *vdev,
bool weak_barriers,
bool context,
@@ -963,19 +1605,20 @@ struct virtqueue *__vring_new_virtqueue(unsigned int index,
void (*callback)(struct virtqueue *),
const char *name)
{
- unsigned int i;
+ unsigned int num, i;
struct vring_virtqueue *vq;
- vq = kmalloc(sizeof(*vq) + vring.num * sizeof(struct vring_desc_state),
+ num = packed ? vring.vring_packed.num : vring.vring_split.num;
+
+ vq = kmalloc(sizeof(*vq) + num * sizeof(struct vring_desc_state),
GFP_KERNEL);
if (!vq)
return NULL;
- vq->vring = vring;
vq->vq.callback = callback;
vq->vq.vdev = vdev;
vq->vq.name = name;
- vq->vq.num_free = vring.num;
+ vq->vq.num_free = num;
vq->vq.index = index;
vq->we_own_ring = false;
vq->queue_dma_addr = 0;
@@ -984,9 +1627,8 @@ struct virtqueue *__vring_new_virtqueue(unsigned int index,
vq->weak_barriers = weak_barriers;
vq->broken = false;
vq->last_used_idx = 0;
- vq->avail_flags_shadow = 0;
- vq->avail_idx_shadow = 0;
vq->num_added = 0;
+ vq->packed = packed;
list_add_tail(&vq->vq.list, &vdev->vqs);
#ifdef DEBUG
vq->in_use = false;
@@ -997,18 +1639,37 @@ struct virtqueue *__vring_new_virtqueue(unsigned int index,
!context;
vq->event = virtio_has_feature(vdev, VIRTIO_RING_F_EVENT_IDX);
+ if (vq->packed) {
+ vq->vring_packed = vring.vring_packed;
+ vq->next_avail_idx = 0;
+ vq->wrap_counter = 1;
+ vq->event_flags_shadow = 0;
+ } else {
+ vq->vring = vring.vring_split;
+ vq->avail_flags_shadow = 0;
+ vq->avail_idx_shadow = 0;
+
+ /* Put everything in free lists. */
+ vq->free_head = 0;
+ for (i = 0; i < num-1; i++)
+ vq->vring.desc[i].next = cpu_to_virtio16(vdev, i + 1);
+ }
+
/* No callback? Tell other side not to bother us. */
if (!callback) {
- vq->avail_flags_shadow |= VRING_AVAIL_F_NO_INTERRUPT;
- if (!vq->event)
- vq->vring.avail->flags = cpu_to_virtio16(vdev, vq->avail_flags_shadow);
+ if (packed) {
+ vq->event_flags_shadow = VRING_EVENT_F_DISABLE;
+ vq->vring_packed.driver->flags = cpu_to_virtio16(vdev,
+ vq->event_flags_shadow);
+ } else {
+ vq->avail_flags_shadow |= VRING_AVAIL_F_NO_INTERRUPT;
+ if (!vq->event)
+ vq->vring.avail->flags = cpu_to_virtio16(vdev,
+ vq->avail_flags_shadow);
+ }
}
- /* Put everything in free lists. */
- vq->free_head = 0;
- for (i = 0; i < vring.num-1; i++)
- vq->vring.desc[i].next = cpu_to_virtio16(vdev, i + 1);
- memset(vq->desc_state, 0, vring.num * sizeof(struct vring_desc_state));
+ memset(vq->desc_state, 0, num * sizeof(struct vring_desc_state));
return &vq->vq;
}
@@ -1056,6 +1717,22 @@ static void vring_free_queue(struct virtio_device *vdev, size_t size,
}
}
+static inline int
+__vring_size(unsigned int num, unsigned long align, bool packed)
+{
+ return packed ? vring_packed_size(num, align) : vring_size(num, align);
+}
+
+static inline void vring_packed_init(struct vring_packed *vr, unsigned int num,
+ void *p, unsigned long align)
+{
+ vr->num = num;
+ vr->desc = p;
+ vr->driver = (void *)(((uintptr_t)p + sizeof(struct vring_packed_desc)
+ * num + align - 1) & ~(align - 1));
+ vr->device = vr->driver + 1;
+}
+
struct virtqueue *vring_create_virtqueue(
unsigned int index,
unsigned int num,
@@ -1072,7 +1749,8 @@ struct virtqueue *vring_create_virtqueue(
void *queue = NULL;
dma_addr_t dma_addr;
size_t queue_size_in_bytes;
- struct vring vring;
+ union vring_union vring;
+ bool packed;
/* We assume num is a power of 2. */
if (num & (num - 1)) {
@@ -1080,9 +1758,13 @@ struct virtqueue *vring_create_virtqueue(
return NULL;
}
+ packed = virtio_has_feature(vdev, VIRTIO_F_RING_PACKED);
+
/* TODO: allocate each queue chunk individually */
- for (; num && vring_size(num, vring_align) > PAGE_SIZE; num /= 2) {
- queue = vring_alloc_queue(vdev, vring_size(num, vring_align),
+ for (; num && __vring_size(num, vring_align, packed) > PAGE_SIZE;
+ num /= 2) {
+ queue = vring_alloc_queue(vdev, __vring_size(num, vring_align,
+ packed),
&dma_addr,
GFP_KERNEL|__GFP_NOWARN|__GFP_ZERO);
if (queue)
@@ -1094,17 +1776,21 @@ struct virtqueue *vring_create_virtqueue(
if (!queue) {
/* Try to get a single page. You are my only hope! */
- queue = vring_alloc_queue(vdev, vring_size(num, vring_align),
+ queue = vring_alloc_queue(vdev, __vring_size(num, vring_align,
+ packed),
&dma_addr, GFP_KERNEL|__GFP_ZERO);
}
if (!queue)
return NULL;
- queue_size_in_bytes = vring_size(num, vring_align);
- vring_init(&vring, num, queue, vring_align);
+ queue_size_in_bytes = __vring_size(num, vring_align, packed);
+ if (packed)
+ vring_packed_init(&vring.vring_packed, num, queue, vring_align);
+ else
+ vring_init(&vring.vring_split, num, queue, vring_align);
- vq = __vring_new_virtqueue(index, vring, vdev, weak_barriers, context,
- notify, callback, name);
+ vq = __vring_new_virtqueue(index, vring, packed, vdev, weak_barriers,
+ context, notify, callback, name);
if (!vq) {
vring_free_queue(vdev, queue_size_in_bytes, queue,
dma_addr);
@@ -1130,10 +1816,17 @@ struct virtqueue *vring_new_virtqueue(unsigned int index,
void (*callback)(struct virtqueue *vq),
const char *name)
{
- struct vring vring;
- vring_init(&vring, num, pages, vring_align);
- return __vring_new_virtqueue(index, vring, vdev, weak_barriers, context,
- notify, callback, name);
+ union vring_union vring;
+ bool packed;
+
+ packed = virtio_has_feature(vdev, VIRTIO_F_RING_PACKED);
+ if (packed)
+ vring_packed_init(&vring.vring_packed, num, pages, vring_align);
+ else
+ vring_init(&vring.vring_split, num, pages, vring_align);
+
+ return __vring_new_virtqueue(index, vring, packed, vdev, weak_barriers,
+ context, notify, callback, name);
}
EXPORT_SYMBOL_GPL(vring_new_virtqueue);
@@ -1143,7 +1836,9 @@ void vring_del_virtqueue(struct virtqueue *_vq)
if (vq->we_own_ring) {
vring_free_queue(vq->vq.vdev, vq->queue_size_in_bytes,
- vq->vring.desc, vq->queue_dma_addr);
+ vq->packed ? (void *)vq->vring_packed.desc :
+ (void *)vq->vring.desc,
+ vq->queue_dma_addr);
}
list_del(&_vq->list);
kfree(vq);
@@ -1157,14 +1852,18 @@ void vring_transport_features(struct virtio_device *vdev)
for (i = VIRTIO_TRANSPORT_F_START; i < VIRTIO_TRANSPORT_F_END; i++) {
switch (i) {
- case VIRTIO_RING_F_INDIRECT_DESC:
+#if 0
+ case VIRTIO_RING_F_INDIRECT_DESC: // FIXME not tested yet.
break;
- case VIRTIO_RING_F_EVENT_IDX:
+ case VIRTIO_RING_F_EVENT_IDX: // FIXME probably not work.
break;
+#endif
case VIRTIO_F_VERSION_1:
break;
case VIRTIO_F_IOMMU_PLATFORM:
break;
+ case VIRTIO_F_RING_PACKED:
+ break;
default:
/* We don't understand this bit. */
__virtio_clear_bit(vdev, i);
@@ -1185,7 +1884,7 @@ unsigned int virtqueue_get_vring_size(struct virtqueue *_vq)
struct vring_virtqueue *vq = to_vvq(_vq);
- return vq->vring.num;
+ return vq->packed ? vq->vring_packed.num : vq->vring.num;
}
EXPORT_SYMBOL_GPL(virtqueue_get_vring_size);
@@ -1228,6 +1927,10 @@ dma_addr_t virtqueue_get_avail_addr(struct virtqueue *_vq)
BUG_ON(!vq->we_own_ring);
+ if (vq->packed)
+ return vq->queue_dma_addr + ((char *)vq->vring_packed.driver -
+ (char *)vq->vring_packed.desc);
+
return vq->queue_dma_addr +
((char *)vq->vring.avail - (char *)vq->vring.desc);
}
@@ -1239,11 +1942,16 @@ dma_addr_t virtqueue_get_used_addr(struct virtqueue *_vq)
BUG_ON(!vq->we_own_ring);
+ if (vq->packed)
+ return vq->queue_dma_addr + ((char *)vq->vring_packed.device -
+ (char *)vq->vring_packed.desc);
+
return vq->queue_dma_addr +
((char *)vq->vring.used - (char *)vq->vring.desc);
}
EXPORT_SYMBOL_GPL(virtqueue_get_used_addr);
+/* Only available for split ring */
const struct vring *virtqueue_get_vring(struct virtqueue *vq)
{
return &to_vvq(vq)->vring;
diff --git a/include/linux/virtio_ring.h b/include/linux/virtio_ring.h
index bbf32524ab27..a0075894ad16 100644
--- a/include/linux/virtio_ring.h
+++ b/include/linux/virtio_ring.h
@@ -60,6 +60,11 @@ static inline void virtio_store_mb(bool weak_barriers,
struct virtio_device;
struct virtqueue;
+union vring_union {
+ struct vring vring_split;
+ struct vring_packed vring_packed;
+};
+
/*
* Creates a virtqueue and allocates the descriptor ring. If
* may_reduce_num is set, then this may allocate a smaller ring than
@@ -79,7 +84,8 @@ struct virtqueue *vring_create_virtqueue(unsigned int index,
/* Creates a virtqueue with a custom layout. */
struct virtqueue *__vring_new_virtqueue(unsigned int index,
- struct vring vring,
+ union vring_union vring,
+ bool packed,
struct virtio_device *vdev,
bool weak_barriers,
bool ctx,
diff --git a/include/uapi/linux/virtio_config.h b/include/uapi/linux/virtio_config.h
index 308e2096291f..a6e392325e3a 100644
--- a/include/uapi/linux/virtio_config.h
+++ b/include/uapi/linux/virtio_config.h
@@ -49,7 +49,7 @@
* transport being used (eg. virtio_ring), the rest are per-device feature
* bits. */
#define VIRTIO_TRANSPORT_F_START 28
-#define VIRTIO_TRANSPORT_F_END 34
+#define VIRTIO_TRANSPORT_F_END 36
#ifndef VIRTIO_CONFIG_NO_LEGACY
/* Do we get callbacks when the ring is completely used, even if we've
@@ -71,4 +71,14 @@
* this is for compatibility with legacy systems.
*/
#define VIRTIO_F_IOMMU_PLATFORM 33
+
+/* This feature indicates support for the packed virtqueue layout. */
+#define VIRTIO_F_RING_PACKED 34
+
+/*
+ * This feature indicates that all buffers are used by the device
+ * in the same order in which they have been made available.
+ */
+#define VIRTIO_F_IN_ORDER 35
+
#endif /* _UAPI_LINUX_VIRTIO_CONFIG_H */
diff --git a/include/uapi/linux/virtio_ring.h b/include/uapi/linux/virtio_ring.h
index 6d5d5faa989b..735d4207c988 100644
--- a/include/uapi/linux/virtio_ring.h
+++ b/include/uapi/linux/virtio_ring.h
@@ -44,6 +44,9 @@
/* This means the buffer contains a list of buffer descriptors. */
#define VRING_DESC_F_INDIRECT 4
+#define VRING_DESC_F_AVAIL(b) ((b) << 7)
+#define VRING_DESC_F_USED(b) ((b) << 15)
+
/* The Host uses this in used->flags to advise the Guest: don't kick me when
* you add a buffer. It's unreliable, so it's simply an optimization. Guest
* will still kick if it's out of buffers. */
@@ -53,6 +56,10 @@
* optimization. */
#define VRING_AVAIL_F_NO_INTERRUPT 1
+#define VRING_EVENT_F_ENABLE 0x0
+#define VRING_EVENT_F_DISABLE 0x1
+#define VRING_EVENT_F_DESC 0x2
+
/* We support indirect buffer descriptors */
#define VIRTIO_RING_F_INDIRECT_DESC 28
@@ -171,4 +178,58 @@ static inline int vring_need_event(__u16 event_idx, __u16 new_idx, __u16 old)
return (__u16)(new_idx - event_idx - 1) < (__u16)(new_idx - old);
}
+struct vring_packed_desc_event {
+ /* __virtio16 off : 15; // Descriptor Event Offset
+ * __virtio16 wrap : 1; // Descriptor Event Wrap Counter */
+ __virtio16 off_wrap;
+ /* __virtio16 flags : 2; // Descriptor Event Flags */
+ __virtio16 flags;
+};
+
+struct vring_packed_desc {
+ /* Buffer Address. */
+ __virtio64 addr;
+ /* Buffer Length. */
+ __virtio32 len;
+ /* Buffer ID. */
+ __virtio16 id;
+ /* The flags depending on descriptor type. */
+ __virtio16 flags;
+};
+
+struct vring_packed {
+ unsigned int num;
+
+ struct vring_packed_desc *desc;
+
+ struct vring_packed_desc_event *driver;
+
+ struct vring_packed_desc_event *device;
+};
+
+/* The standard layout for the packed ring is a continuous chunk of memory
+ * which looks like this.
+ *
+ * struct vring_packed
+ * {
+ * // The actual descriptors (16 bytes each)
+ * struct vring_packed_desc desc[num];
+ *
+ * // Padding to the next align boundary.
+ * char pad[];
+ *
+ * // Driver Event Suppression
+ * struct vring_packed_desc_event driver;
+ *
+ * // Device Event Suppression
+ * struct vring_packed_desc_event device;
+ * };
+ */
+
+static inline unsigned vring_packed_size(unsigned int num, unsigned long align)
+{
+ return ((sizeof(struct vring_packed_desc) * num + align - 1)
+ & ~(align - 1)) + sizeof(struct vring_packed_desc_event) * 2;
+}
+
#endif /* _UAPI_LINUX_VIRTIO_RING_H */
--
2.11.0
^ permalink raw reply related
* [PATCH v2 net-next 00/10] mlxsw: Various cleanups
From: Ido Schimmel @ 2018-04-01 14:34 UTC (permalink / raw)
To: netdev; +Cc: davem, jiri, petrm, dsahern, mlxsw, Ido Schimmel
Hi,
The first nine patches from Jiri perform small and unrelated cleanups.
The largest being the conversion of the KVD linear partitions from a
list to an array, which simplifies the code.
The last patch from Petr is a bug fix for a recent net-next commit that
prevented the "kvd" resource from being marked as the parent of its
various child resources (e.g., "/kvd/linear").
v2: Dropped devlink patch following David's comment. Will be sent
separately.
Jiri Pirko (9):
mlxsw: spectrum_acl: Fix flex actions header ifndef define construct
mlxsw: spectrum_kvdl: Fix handling of resource_size_param
mlxsw: Constify devlink_resource_ops
mlxsw: spectrum: Change KVD linear parts from list to array
mlxsw: remove kvd_hash_granularity from config profile struct
mlxsw: core: Fix arg name of MLXSW_CORE_RES_VALID and
MLXSW_CORE_RES_GET
mlxsw: Move "used_kvd_sizes" check to mlxsw_pci_config_profile
mlxsw: Move "resources_query_enable" out of mlxsw_config_profile
mlxsw: spectrum: Pass mlxsw_core as arg of
mlxsw_sp_kvdl_resources_register()
Petr Machata (1):
mlxsw: spectrum: Don't use resource ID of 0
drivers/net/ethernet/mellanox/mlxsw/core.c | 5 +-
drivers/net/ethernet/mellanox/mlxsw/core.h | 14 +-
drivers/net/ethernet/mellanox/mlxsw/pci.c | 11 +-
drivers/net/ethernet/mellanox/mlxsw/spectrum.c | 16 +-
drivers/net/ethernet/mellanox/mlxsw/spectrum.h | 4 +-
.../mellanox/mlxsw/spectrum_acl_flex_actions.h | 4 +-
.../net/ethernet/mellanox/mlxsw/spectrum_kvdl.c | 284 ++++++++-------------
drivers/net/ethernet/mellanox/mlxsw/switchib.c | 1 -
drivers/net/ethernet/mellanox/mlxsw/switchx2.c | 1 -
9 files changed, 131 insertions(+), 209 deletions(-)
--
2.14.3
^ permalink raw reply
* [PATCH v2 net-next 01/10] mlxsw: spectrum_acl: Fix flex actions header ifndef define construct
From: Ido Schimmel @ 2018-04-01 14:34 UTC (permalink / raw)
To: netdev; +Cc: davem, jiri, petrm, dsahern, mlxsw, Ido Schimmel
In-Reply-To: <20180401143459.30770-1-idosch@mellanox.com>
From: Jiri Pirko <jiri@mellanox.com>
Fix copy&paste error in flex actions header ifndef define construct
Signed-off-by: Jiri Pirko <jiri@mellanox.com>
Signed-off-by: Ido Schimmel <idosch@mellanox.com>
---
drivers/net/ethernet/mellanox/mlxsw/spectrum_acl_flex_actions.h | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlxsw/spectrum_acl_flex_actions.h b/drivers/net/ethernet/mellanox/mlxsw/spectrum_acl_flex_actions.h
index 2726192836ad..bd6d552d95b9 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/spectrum_acl_flex_actions.h
+++ b/drivers/net/ethernet/mellanox/mlxsw/spectrum_acl_flex_actions.h
@@ -33,8 +33,8 @@
* POSSIBILITY OF SUCH DAMAGE.
*/
-#ifndef _MLXSW_SPECTRUM_ACL_FLEX_KEYS_H
-#define _MLXSW_SPECTRUM_ACL_FLEX_KEYS_H
+#ifndef _MLXSW_SPECTRUM_ACL_FLEX_ACTIONS_H
+#define _MLXSW_SPECTRUM_ACL_FLEX_ACTIONS_H
#include "spectrum.h"
--
2.14.3
^ permalink raw reply related
* [PATCH v2 net-next 02/10] mlxsw: spectrum_kvdl: Fix handling of resource_size_param
From: Ido Schimmel @ 2018-04-01 14:34 UTC (permalink / raw)
To: netdev; +Cc: davem, jiri, petrm, dsahern, mlxsw, Ido Schimmel
In-Reply-To: <20180401143459.30770-1-idosch@mellanox.com>
From: Jiri Pirko <jiri@mellanox.com>
Current code uses global variables, adjusts them and passes pointer down
to devlink. With every other mlxsw_core instance, the previously passed
pointer values are rewritten. Fix this by de-globalize the variables.
Fixes: 7f47b19bd744 ("mlxsw: spectrum_kvdl: Add support for per part occupancy")
Signed-off-by: Jiri Pirko <jiri@mellanox.com>
Acked-by: Arkadi Sharshevsky <arkadis@mellanox.com>
Signed-off-by: Ido Schimmel <idosch@mellanox.com>
---
.../net/ethernet/mellanox/mlxsw/spectrum_kvdl.c | 47 +++++++---------------
1 file changed, 14 insertions(+), 33 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlxsw/spectrum_kvdl.c b/drivers/net/ethernet/mellanox/mlxsw/spectrum_kvdl.c
index 85503e93b93f..9e61518c4945 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/spectrum_kvdl.c
+++ b/drivers/net/ethernet/mellanox/mlxsw/spectrum_kvdl.c
@@ -420,67 +420,48 @@ static struct devlink_resource_ops mlxsw_sp_kvdl_chunks_large_ops = {
.occ_get = mlxsw_sp_kvdl_large_chunks_occ_get,
};
-static struct devlink_resource_size_params mlxsw_sp_kvdl_single_size_params = {
- .size_min = 0,
- .size_granularity = 1,
- .unit = DEVLINK_RESOURCE_UNIT_ENTRY,
-};
-
-static struct devlink_resource_size_params mlxsw_sp_kvdl_chunks_size_params = {
- .size_min = 0,
- .size_granularity = MLXSW_SP_CHUNK_MAX,
- .unit = DEVLINK_RESOURCE_UNIT_ENTRY,
-};
-
-static struct devlink_resource_size_params mlxsw_sp_kvdl_large_chunks_size_params = {
- .size_min = 0,
- .size_granularity = MLXSW_SP_LARGE_CHUNK_MAX,
- .unit = DEVLINK_RESOURCE_UNIT_ENTRY,
-};
-
-static void
-mlxsw_sp_kvdl_resource_size_params_prepare(struct devlink *devlink)
+int mlxsw_sp_kvdl_resources_register(struct devlink *devlink)
{
struct mlxsw_core *mlxsw_core = devlink_priv(devlink);
+ static struct devlink_resource_size_params size_params;
u32 kvdl_max_size;
+ int err;
kvdl_max_size = MLXSW_CORE_RES_GET(mlxsw_core, KVD_SIZE) -
MLXSW_CORE_RES_GET(mlxsw_core, KVD_SINGLE_MIN_SIZE) -
MLXSW_CORE_RES_GET(mlxsw_core, KVD_DOUBLE_MIN_SIZE);
- mlxsw_sp_kvdl_single_size_params.size_max = kvdl_max_size;
- mlxsw_sp_kvdl_chunks_size_params.size_max = kvdl_max_size;
- mlxsw_sp_kvdl_large_chunks_size_params.size_max = kvdl_max_size;
-}
-
-int mlxsw_sp_kvdl_resources_register(struct devlink *devlink)
-{
- int err;
-
- mlxsw_sp_kvdl_resource_size_params_prepare(devlink);
+ devlink_resource_size_params_init(&size_params, 0, kvdl_max_size, 1,
+ DEVLINK_RESOURCE_UNIT_ENTRY);
err = devlink_resource_register(devlink, MLXSW_SP_RESOURCE_NAME_KVD_LINEAR_SINGLES,
MLXSW_SP_KVDL_SINGLE_SIZE,
MLXSW_SP_RESOURCE_KVD_LINEAR_SINGLE,
MLXSW_SP_RESOURCE_KVD_LINEAR,
- &mlxsw_sp_kvdl_single_size_params,
+ &size_params,
&mlxsw_sp_kvdl_single_ops);
if (err)
return err;
+ devlink_resource_size_params_init(&size_params, 0, kvdl_max_size,
+ MLXSW_SP_CHUNK_MAX,
+ DEVLINK_RESOURCE_UNIT_ENTRY);
err = devlink_resource_register(devlink, MLXSW_SP_RESOURCE_NAME_KVD_LINEAR_CHUNKS,
MLXSW_SP_KVDL_CHUNKS_SIZE,
MLXSW_SP_RESOURCE_KVD_LINEAR_CHUNKS,
MLXSW_SP_RESOURCE_KVD_LINEAR,
- &mlxsw_sp_kvdl_chunks_size_params,
+ &size_params,
&mlxsw_sp_kvdl_chunks_ops);
if (err)
return err;
+ devlink_resource_size_params_init(&size_params, 0, kvdl_max_size,
+ MLXSW_SP_LARGE_CHUNK_MAX,
+ DEVLINK_RESOURCE_UNIT_ENTRY);
err = devlink_resource_register(devlink, MLXSW_SP_RESOURCE_NAME_KVD_LINEAR_LARGE_CHUNKS,
MLXSW_SP_KVDL_LARGE_CHUNKS_SIZE,
MLXSW_SP_RESOURCE_KVD_LINEAR_LARGE_CHUNKS,
MLXSW_SP_RESOURCE_KVD_LINEAR,
- &mlxsw_sp_kvdl_large_chunks_size_params,
+ &size_params,
&mlxsw_sp_kvdl_chunks_large_ops);
return err;
}
--
2.14.3
^ permalink raw reply related
* [PATCH v2 net-next 03/10] mlxsw: Constify devlink_resource_ops
From: Ido Schimmel @ 2018-04-01 14:34 UTC (permalink / raw)
To: netdev; +Cc: davem, jiri, petrm, dsahern, mlxsw, Ido Schimmel
In-Reply-To: <20180401143459.30770-1-idosch@mellanox.com>
From: Jiri Pirko <jiri@mellanox.com>
devlink_resource_ops should be const as the arg of register function is
also const.
Signed-off-by: Jiri Pirko <jiri@mellanox.com>
Signed-off-by: Ido Schimmel <idosch@mellanox.com>
---
drivers/net/ethernet/mellanox/mlxsw/spectrum.c | 2 +-
drivers/net/ethernet/mellanox/mlxsw/spectrum_kvdl.c | 6 +++---
2 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlxsw/spectrum.c b/drivers/net/ethernet/mellanox/mlxsw/spectrum.c
index 4aa84442e357..0e9ed41ce8bc 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/spectrum.c
+++ b/drivers/net/ethernet/mellanox/mlxsw/spectrum.c
@@ -3815,7 +3815,7 @@ static u64 mlxsw_sp_resource_kvd_linear_occ_get(struct devlink *devlink)
return mlxsw_sp_kvdl_occ_get(mlxsw_sp);
}
-static struct devlink_resource_ops mlxsw_sp_resource_kvd_linear_ops = {
+static const struct devlink_resource_ops mlxsw_sp_resource_kvd_linear_ops = {
.occ_get = mlxsw_sp_resource_kvd_linear_occ_get,
};
diff --git a/drivers/net/ethernet/mellanox/mlxsw/spectrum_kvdl.c b/drivers/net/ethernet/mellanox/mlxsw/spectrum_kvdl.c
index 9e61518c4945..201825c0019b 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/spectrum_kvdl.c
+++ b/drivers/net/ethernet/mellanox/mlxsw/spectrum_kvdl.c
@@ -408,15 +408,15 @@ static u64 mlxsw_sp_kvdl_large_chunks_occ_get(struct devlink *devlink)
return mlxsw_sp_kvdl_part_occ(part);
}
-static struct devlink_resource_ops mlxsw_sp_kvdl_single_ops = {
+static const struct devlink_resource_ops mlxsw_sp_kvdl_single_ops = {
.occ_get = mlxsw_sp_kvdl_single_occ_get,
};
-static struct devlink_resource_ops mlxsw_sp_kvdl_chunks_ops = {
+static const struct devlink_resource_ops mlxsw_sp_kvdl_chunks_ops = {
.occ_get = mlxsw_sp_kvdl_chunks_occ_get,
};
-static struct devlink_resource_ops mlxsw_sp_kvdl_chunks_large_ops = {
+static const struct devlink_resource_ops mlxsw_sp_kvdl_chunks_large_ops = {
.occ_get = mlxsw_sp_kvdl_large_chunks_occ_get,
};
--
2.14.3
^ permalink raw reply related
* [PATCH v2 net-next 04/10] mlxsw: spectrum: Change KVD linear parts from list to array
From: Ido Schimmel @ 2018-04-01 14:34 UTC (permalink / raw)
To: netdev; +Cc: davem, jiri, petrm, dsahern, mlxsw, Ido Schimmel
In-Reply-To: <20180401143459.30770-1-idosch@mellanox.com>
From: Jiri Pirko <jiri@mellanox.com>
The parts info is array. The parts copy this info array, yet they are a
list. So make the indexing according to the id and change the list of
parts into array of parts. This helps to eliminate lookups and
constructs like mlxsw_sp_kvdl_part_update() (took me some non-trivial
time to figure out what is going on there).
Alongside with that, introduce a helper macro to define the parts infos.
Signed-off-by: Jiri Pirko <jiri@mellanox.com>
Signed-off-by: Ido Schimmel <idosch@mellanox.com>
---
.../net/ethernet/mellanox/mlxsw/spectrum_kvdl.c | 235 ++++++++-------------
1 file changed, 92 insertions(+), 143 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlxsw/spectrum_kvdl.c b/drivers/net/ethernet/mellanox/mlxsw/spectrum_kvdl.c
index 201825c0019b..7b28f65d6407 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/spectrum_kvdl.c
+++ b/drivers/net/ethernet/mellanox/mlxsw/spectrum_kvdl.c
@@ -55,24 +55,47 @@
#define MLXSW_SP_KVDL_LARGE_CHUNKS_END \
(MLXSW_SP_KVDL_LARGE_CHUNKS_SIZE + MLXSW_SP_KVDL_LARGE_CHUNKS_BASE - 1)
-#define MLXSW_SP_CHUNK_MAX 32
-#define MLXSW_SP_LARGE_CHUNK_MAX 512
+#define MLXSW_SP_KVDL_SINGLE_ALLOC_SIZE 1
+#define MLXSW_SP_KVDL_CHUNKS_ALLOC_SIZE 32
+#define MLXSW_SP_KVDL_LARGE_CHUNKS_ALLOC_SIZE 512
struct mlxsw_sp_kvdl_part_info {
unsigned int part_index;
unsigned int start_index;
unsigned int end_index;
unsigned int alloc_size;
+ enum mlxsw_sp_resource_id resource_id;
+};
+
+enum mlxsw_sp_kvdl_part_id {
+ MLXSW_SP_KVDL_PART_ID_SINGLE,
+ MLXSW_SP_KVDL_PART_ID_CHUNKS,
+ MLXSW_SP_KVDL_PART_ID_LARGE_CHUNKS,
};
+#define MLXSW_SP_KVDL_PART_INFO(id) \
+[MLXSW_SP_KVDL_PART_ID_##id] = { \
+ .start_index = MLXSW_SP_KVDL_##id##_BASE, \
+ .end_index = MLXSW_SP_KVDL_##id##_END, \
+ .alloc_size = MLXSW_SP_KVDL_##id##_ALLOC_SIZE, \
+ .resource_id = MLXSW_SP_RESOURCE_KVD_LINEAR_##id, \
+}
+
+static const struct mlxsw_sp_kvdl_part_info mlxsw_sp_kvdl_parts_info[] = {
+ MLXSW_SP_KVDL_PART_INFO(SINGLE),
+ MLXSW_SP_KVDL_PART_INFO(CHUNKS),
+ MLXSW_SP_KVDL_PART_INFO(LARGE_CHUNKS),
+};
+
+#define MLXSW_SP_KVDL_PARTS_INFO_LEN ARRAY_SIZE(mlxsw_sp_kvdl_parts_info)
+
struct mlxsw_sp_kvdl_part {
- struct list_head list;
- struct mlxsw_sp_kvdl_part_info *info;
+ struct mlxsw_sp_kvdl_part_info info;
unsigned long usage[0]; /* Entries */
};
struct mlxsw_sp_kvdl {
- struct list_head parts_list;
+ struct mlxsw_sp_kvdl_part *parts[MLXSW_SP_KVDL_PARTS_INFO_LEN];
};
static struct mlxsw_sp_kvdl_part *
@@ -80,11 +103,13 @@ mlxsw_sp_kvdl_alloc_size_part(struct mlxsw_sp_kvdl *kvdl,
unsigned int alloc_size)
{
struct mlxsw_sp_kvdl_part *part, *min_part = NULL;
+ int i;
- list_for_each_entry(part, &kvdl->parts_list, list) {
- if (alloc_size <= part->info->alloc_size &&
+ for (i = 0; i < MLXSW_SP_KVDL_PARTS_INFO_LEN; i++) {
+ part = kvdl->parts[i];
+ if (alloc_size <= part->info.alloc_size &&
(!min_part ||
- part->info->alloc_size <= min_part->info->alloc_size))
+ part->info.alloc_size <= min_part->info.alloc_size))
min_part = part;
}
@@ -95,10 +120,12 @@ static struct mlxsw_sp_kvdl_part *
mlxsw_sp_kvdl_index_part(struct mlxsw_sp_kvdl *kvdl, u32 kvdl_index)
{
struct mlxsw_sp_kvdl_part *part;
+ int i;
- list_for_each_entry(part, &kvdl->parts_list, list) {
- if (kvdl_index >= part->info->start_index &&
- kvdl_index <= part->info->end_index)
+ for (i = 0; i < MLXSW_SP_KVDL_PARTS_INFO_LEN; i++) {
+ part = kvdl->parts[i];
+ if (kvdl_index >= part->info.start_index &&
+ kvdl_index <= part->info.end_index)
return part;
}
@@ -122,7 +149,7 @@ mlxsw_sp_kvdl_index_entry_index(const struct mlxsw_sp_kvdl_part_info *info,
static int mlxsw_sp_kvdl_part_alloc(struct mlxsw_sp_kvdl_part *part,
u32 *p_kvdl_index)
{
- const struct mlxsw_sp_kvdl_part_info *info = part->info;
+ const struct mlxsw_sp_kvdl_part_info *info = &part->info;
unsigned int entry_index, nr_entries;
nr_entries = (info->end_index - info->start_index + 1) /
@@ -132,8 +159,7 @@ static int mlxsw_sp_kvdl_part_alloc(struct mlxsw_sp_kvdl_part *part,
return -ENOBUFS;
__set_bit(entry_index, part->usage);
- *p_kvdl_index = mlxsw_sp_entry_index_kvdl_index(part->info,
- entry_index);
+ *p_kvdl_index = mlxsw_sp_entry_index_kvdl_index(info, entry_index);
return 0;
}
@@ -141,10 +167,10 @@ static int mlxsw_sp_kvdl_part_alloc(struct mlxsw_sp_kvdl_part *part,
static void mlxsw_sp_kvdl_part_free(struct mlxsw_sp_kvdl_part *part,
u32 kvdl_index)
{
+ const struct mlxsw_sp_kvdl_part_info *info = &part->info;
unsigned int entry_index;
- entry_index = mlxsw_sp_kvdl_index_entry_index(part->info,
- kvdl_index);
+ entry_index = mlxsw_sp_kvdl_index_entry_index(info, kvdl_index);
__clear_bit(entry_index, part->usage);
}
@@ -183,74 +209,30 @@ int mlxsw_sp_kvdl_alloc_size_query(struct mlxsw_sp *mlxsw_sp,
if (IS_ERR(part))
return PTR_ERR(part);
- *p_alloc_size = part->info->alloc_size;
+ *p_alloc_size = part->info.alloc_size;
return 0;
}
-enum mlxsw_sp_kvdl_part_id {
- MLXSW_SP_KVDL_PART_SINGLE,
- MLXSW_SP_KVDL_PART_CHUNKS,
- MLXSW_SP_KVDL_PART_LARGE_CHUNKS,
-};
-
-static const struct mlxsw_sp_kvdl_part_info kvdl_parts_info[] = {
- {
- .part_index = MLXSW_SP_KVDL_PART_SINGLE,
- .start_index = MLXSW_SP_KVDL_SINGLE_BASE,
- .end_index = MLXSW_SP_KVDL_SINGLE_END,
- .alloc_size = 1,
- },
- {
- .part_index = MLXSW_SP_KVDL_PART_CHUNKS,
- .start_index = MLXSW_SP_KVDL_CHUNKS_BASE,
- .end_index = MLXSW_SP_KVDL_CHUNKS_END,
- .alloc_size = MLXSW_SP_CHUNK_MAX,
- },
- {
- .part_index = MLXSW_SP_KVDL_PART_LARGE_CHUNKS,
- .start_index = MLXSW_SP_KVDL_LARGE_CHUNKS_BASE,
- .end_index = MLXSW_SP_KVDL_LARGE_CHUNKS_END,
- .alloc_size = MLXSW_SP_LARGE_CHUNK_MAX,
- },
-};
-
-static struct mlxsw_sp_kvdl_part *
-mlxsw_sp_kvdl_part_find(struct mlxsw_sp *mlxsw_sp, unsigned int part_index)
-{
- struct mlxsw_sp_kvdl_part *part;
-
- list_for_each_entry(part, &mlxsw_sp->kvdl->parts_list, list) {
- if (part->info->part_index == part_index)
- return part;
- }
-
- return NULL;
-}
-
-static void
-mlxsw_sp_kvdl_part_update(struct mlxsw_sp *mlxsw_sp,
- struct mlxsw_sp_kvdl_part *part, unsigned int size)
+static void mlxsw_sp_kvdl_part_update(struct mlxsw_sp_kvdl_part *part,
+ struct mlxsw_sp_kvdl_part *part_prev,
+ unsigned int size)
{
- struct mlxsw_sp_kvdl_part_info *info = part->info;
-
- if (list_is_last(&part->list, &mlxsw_sp->kvdl->parts_list)) {
- info->end_index = size - 1;
- } else {
- struct mlxsw_sp_kvdl_part *last_part;
- last_part = list_next_entry(part, list);
- info->start_index = last_part->info->end_index + 1;
- info->end_index = info->start_index + size - 1;
+ if (!part_prev) {
+ part->info.end_index = size - 1;
+ } else {
+ part->info.start_index = part_prev->info.end_index + 1;
+ part->info.end_index = part->info.start_index + size - 1;
}
}
-static int mlxsw_sp_kvdl_part_init(struct mlxsw_sp *mlxsw_sp,
- unsigned int part_index)
+static struct mlxsw_sp_kvdl_part *
+mlxsw_sp_kvdl_part_init(struct mlxsw_sp *mlxsw_sp,
+ const struct mlxsw_sp_kvdl_part_info *info,
+ struct mlxsw_sp_kvdl_part *part_prev)
{
struct devlink *devlink = priv_to_devlink(mlxsw_sp->core);
- const struct mlxsw_sp_kvdl_part_info *info;
- enum mlxsw_sp_resource_id resource_id;
struct mlxsw_sp_kvdl_part *part;
bool need_update = true;
unsigned int nr_entries;
@@ -258,23 +240,8 @@ static int mlxsw_sp_kvdl_part_init(struct mlxsw_sp *mlxsw_sp,
u64 resource_size;
int err;
- info = &kvdl_parts_info[part_index];
-
- switch (part_index) {
- case MLXSW_SP_KVDL_PART_SINGLE:
- resource_id = MLXSW_SP_RESOURCE_KVD_LINEAR_SINGLE;
- break;
- case MLXSW_SP_KVDL_PART_CHUNKS:
- resource_id = MLXSW_SP_RESOURCE_KVD_LINEAR_CHUNKS;
- break;
- case MLXSW_SP_KVDL_PART_LARGE_CHUNKS:
- resource_id = MLXSW_SP_RESOURCE_KVD_LINEAR_LARGE_CHUNKS;
- break;
- default:
- return -EINVAL;
- }
-
- err = devlink_resource_size_get(devlink, resource_id, &resource_size);
+ err = devlink_resource_size_get(devlink, info->resource_id,
+ &resource_size);
if (err) {
need_update = false;
resource_size = info->end_index - info->start_index + 1;
@@ -284,86 +251,77 @@ static int mlxsw_sp_kvdl_part_init(struct mlxsw_sp *mlxsw_sp,
usage_size = BITS_TO_LONGS(nr_entries) * sizeof(unsigned long);
part = kzalloc(sizeof(*part) + usage_size, GFP_KERNEL);
if (!part)
- return -ENOMEM;
+ return ERR_PTR(-ENOMEM);
- part->info = kmemdup(info, sizeof(*part->info), GFP_KERNEL);
- if (!part->info)
- goto err_part_info_alloc;
+ memcpy(&part->info, info, sizeof(part->info));
- list_add(&part->list, &mlxsw_sp->kvdl->parts_list);
if (need_update)
- mlxsw_sp_kvdl_part_update(mlxsw_sp, part, resource_size);
- return 0;
-
-err_part_info_alloc:
- kfree(part);
- return -ENOMEM;
+ mlxsw_sp_kvdl_part_update(part, part_prev, resource_size);
+ return part;
}
-static void mlxsw_sp_kvdl_part_fini(struct mlxsw_sp *mlxsw_sp,
- unsigned int part_index)
+static void mlxsw_sp_kvdl_part_fini(struct mlxsw_sp_kvdl_part *part)
{
- struct mlxsw_sp_kvdl_part *part;
-
- part = mlxsw_sp_kvdl_part_find(mlxsw_sp, part_index);
- if (!part)
- return;
-
- list_del(&part->list);
- kfree(part->info);
kfree(part);
}
static int mlxsw_sp_kvdl_parts_init(struct mlxsw_sp *mlxsw_sp)
{
+ struct mlxsw_sp_kvdl *kvdl = mlxsw_sp->kvdl;
+ const struct mlxsw_sp_kvdl_part_info *info;
+ struct mlxsw_sp_kvdl_part *part_prev = NULL;
int err, i;
- INIT_LIST_HEAD(&mlxsw_sp->kvdl->parts_list);
-
- for (i = 0; i < ARRAY_SIZE(kvdl_parts_info); i++) {
- err = mlxsw_sp_kvdl_part_init(mlxsw_sp, i);
- if (err)
+ for (i = 0; i < MLXSW_SP_KVDL_PARTS_INFO_LEN; i++) {
+ info = &mlxsw_sp_kvdl_parts_info[i];
+ kvdl->parts[i] = mlxsw_sp_kvdl_part_init(mlxsw_sp, info,
+ part_prev);
+ if (IS_ERR(kvdl->parts[i])) {
+ err = PTR_ERR(kvdl->parts[i]);
goto err_kvdl_part_init;
+ }
+ part_prev = kvdl->parts[i];
}
-
return 0;
err_kvdl_part_init:
for (i--; i >= 0; i--)
- mlxsw_sp_kvdl_part_fini(mlxsw_sp, i);
+ mlxsw_sp_kvdl_part_fini(kvdl->parts[i]);
return err;
}
static void mlxsw_sp_kvdl_parts_fini(struct mlxsw_sp *mlxsw_sp)
{
+ struct mlxsw_sp_kvdl *kvdl = mlxsw_sp->kvdl;
int i;
- for (i = ARRAY_SIZE(kvdl_parts_info) - 1; i >= 0; i--)
- mlxsw_sp_kvdl_part_fini(mlxsw_sp, i);
+ for (i = 0; i < MLXSW_SP_KVDL_PARTS_INFO_LEN; i++)
+ mlxsw_sp_kvdl_part_fini(kvdl->parts[i]);
}
static u64 mlxsw_sp_kvdl_part_occ(struct mlxsw_sp_kvdl_part *part)
{
+ const struct mlxsw_sp_kvdl_part_info *info = &part->info;
unsigned int nr_entries;
int bit = -1;
u64 occ = 0;
- nr_entries = (part->info->end_index -
- part->info->start_index + 1) /
- part->info->alloc_size;
+ nr_entries = (info->end_index -
+ info->start_index + 1) /
+ info->alloc_size;
while ((bit = find_next_bit(part->usage, nr_entries, bit + 1))
< nr_entries)
- occ += part->info->alloc_size;
+ occ += info->alloc_size;
return occ;
}
u64 mlxsw_sp_kvdl_occ_get(const struct mlxsw_sp *mlxsw_sp)
{
- struct mlxsw_sp_kvdl_part *part;
u64 occ = 0;
+ int i;
- list_for_each_entry(part, &mlxsw_sp->kvdl->parts_list, list)
- occ += mlxsw_sp_kvdl_part_occ(part);
+ for (i = 0; i < MLXSW_SP_KVDL_PARTS_INFO_LEN; i++)
+ occ += mlxsw_sp_kvdl_part_occ(mlxsw_sp->kvdl->parts[i]);
return occ;
}
@@ -374,10 +332,7 @@ static u64 mlxsw_sp_kvdl_single_occ_get(struct devlink *devlink)
struct mlxsw_sp *mlxsw_sp = mlxsw_core_driver_priv(mlxsw_core);
struct mlxsw_sp_kvdl_part *part;
- part = mlxsw_sp_kvdl_part_find(mlxsw_sp, MLXSW_SP_KVDL_PART_SINGLE);
- if (!part)
- return -EINVAL;
-
+ part = mlxsw_sp->kvdl->parts[MLXSW_SP_KVDL_PART_ID_SINGLE];
return mlxsw_sp_kvdl_part_occ(part);
}
@@ -387,10 +342,7 @@ static u64 mlxsw_sp_kvdl_chunks_occ_get(struct devlink *devlink)
struct mlxsw_sp *mlxsw_sp = mlxsw_core_driver_priv(mlxsw_core);
struct mlxsw_sp_kvdl_part *part;
- part = mlxsw_sp_kvdl_part_find(mlxsw_sp, MLXSW_SP_KVDL_PART_CHUNKS);
- if (!part)
- return -EINVAL;
-
+ part = mlxsw_sp->kvdl->parts[MLXSW_SP_KVDL_PART_ID_CHUNKS];
return mlxsw_sp_kvdl_part_occ(part);
}
@@ -400,11 +352,7 @@ static u64 mlxsw_sp_kvdl_large_chunks_occ_get(struct devlink *devlink)
struct mlxsw_sp *mlxsw_sp = mlxsw_core_driver_priv(mlxsw_core);
struct mlxsw_sp_kvdl_part *part;
- part = mlxsw_sp_kvdl_part_find(mlxsw_sp,
- MLXSW_SP_KVDL_PART_LARGE_CHUNKS);
- if (!part)
- return -EINVAL;
-
+ part = mlxsw_sp->kvdl->parts[MLXSW_SP_KVDL_PART_ID_LARGE_CHUNKS];
return mlxsw_sp_kvdl_part_occ(part);
}
@@ -431,7 +379,8 @@ int mlxsw_sp_kvdl_resources_register(struct devlink *devlink)
MLXSW_CORE_RES_GET(mlxsw_core, KVD_SINGLE_MIN_SIZE) -
MLXSW_CORE_RES_GET(mlxsw_core, KVD_DOUBLE_MIN_SIZE);
- devlink_resource_size_params_init(&size_params, 0, kvdl_max_size, 1,
+ devlink_resource_size_params_init(&size_params, 0, kvdl_max_size,
+ MLXSW_SP_KVDL_SINGLE_ALLOC_SIZE,
DEVLINK_RESOURCE_UNIT_ENTRY);
err = devlink_resource_register(devlink, MLXSW_SP_RESOURCE_NAME_KVD_LINEAR_SINGLES,
MLXSW_SP_KVDL_SINGLE_SIZE,
@@ -443,7 +392,7 @@ int mlxsw_sp_kvdl_resources_register(struct devlink *devlink)
return err;
devlink_resource_size_params_init(&size_params, 0, kvdl_max_size,
- MLXSW_SP_CHUNK_MAX,
+ MLXSW_SP_KVDL_CHUNKS_ALLOC_SIZE,
DEVLINK_RESOURCE_UNIT_ENTRY);
err = devlink_resource_register(devlink, MLXSW_SP_RESOURCE_NAME_KVD_LINEAR_CHUNKS,
MLXSW_SP_KVDL_CHUNKS_SIZE,
@@ -455,7 +404,7 @@ int mlxsw_sp_kvdl_resources_register(struct devlink *devlink)
return err;
devlink_resource_size_params_init(&size_params, 0, kvdl_max_size,
- MLXSW_SP_LARGE_CHUNK_MAX,
+ MLXSW_SP_KVDL_LARGE_CHUNKS_ALLOC_SIZE,
DEVLINK_RESOURCE_UNIT_ENTRY);
err = devlink_resource_register(devlink, MLXSW_SP_RESOURCE_NAME_KVD_LINEAR_LARGE_CHUNKS,
MLXSW_SP_KVDL_LARGE_CHUNKS_SIZE,
--
2.14.3
^ permalink raw reply related
* [PATCH v2 net-next 05/10] mlxsw: remove kvd_hash_granularity from config profile struct
From: Ido Schimmel @ 2018-04-01 14:34 UTC (permalink / raw)
To: netdev; +Cc: davem, jiri, petrm, dsahern, mlxsw, Ido Schimmel
In-Reply-To: <20180401143459.30770-1-idosch@mellanox.com>
From: Jiri Pirko <jiri@mellanox.com>
This should not be part of the struct, as the struct fields
are tightly coupled with the FW command payload of the same name.
Just use the "granularity" define directly, as in other places.
Signed-off-by: Jiri Pirko <jiri@mellanox.com>
Signed-off-by: Ido Schimmel <idosch@mellanox.com>
---
drivers/net/ethernet/mellanox/mlxsw/core.h | 1 -
drivers/net/ethernet/mellanox/mlxsw/spectrum.c | 5 ++---
2 files changed, 2 insertions(+), 4 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlxsw/core.h b/drivers/net/ethernet/mellanox/mlxsw/core.h
index 5ddafd74dc00..fd30eaf40475 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/core.h
+++ b/drivers/net/ethernet/mellanox/mlxsw/core.h
@@ -256,7 +256,6 @@ struct mlxsw_config_profile {
u16 adaptive_routing_group_cap;
u8 arn;
u32 kvd_linear_size;
- u16 kvd_hash_granularity;
u8 kvd_hash_single_parts;
u8 kvd_hash_double_parts;
u8 resource_query_enable;
diff --git a/drivers/net/ethernet/mellanox/mlxsw/spectrum.c b/drivers/net/ethernet/mellanox/mlxsw/spectrum.c
index 0e9ed41ce8bc..d503cdbeae29 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/spectrum.c
+++ b/drivers/net/ethernet/mellanox/mlxsw/spectrum.c
@@ -3794,7 +3794,6 @@ static const struct mlxsw_config_profile mlxsw_sp_config_profile = {
.used_max_pkey = 1,
.max_pkey = 0,
.used_kvd_split_data = 1,
- .kvd_hash_granularity = MLXSW_SP_KVD_GRANULARITY,
.kvd_hash_single_parts = 59,
.kvd_hash_double_parts = 41,
.kvd_linear_size = MLXSW_SP_KVD_LINEAR_SIZE,
@@ -3902,7 +3901,7 @@ static int mlxsw_sp_resources_register(struct mlxsw_core *mlxsw_core)
double_size *= profile->kvd_hash_double_parts;
double_size /= profile->kvd_hash_double_parts +
profile->kvd_hash_single_parts;
- double_size = rounddown(double_size, profile->kvd_hash_granularity);
+ double_size = rounddown(double_size, MLXSW_SP_KVD_GRANULARITY);
err = devlink_resource_register(devlink, MLXSW_SP_RESOURCE_NAME_KVD_HASH_DOUBLE,
double_size,
MLXSW_SP_RESOURCE_KVD_HASH_DOUBLE,
@@ -3962,7 +3961,7 @@ static int mlxsw_sp_kvd_sizes_get(struct mlxsw_core *mlxsw_core,
double_size /= profile->kvd_hash_double_parts +
profile->kvd_hash_single_parts;
*p_double_size = rounddown(double_size,
- profile->kvd_hash_granularity);
+ MLXSW_SP_KVD_GRANULARITY);
}
err = devlink_resource_size_get(devlink,
--
2.14.3
^ permalink raw reply related
* [PATCH v2 net-next 06/10] mlxsw: core: Fix arg name of MLXSW_CORE_RES_VALID and MLXSW_CORE_RES_GET
From: Ido Schimmel @ 2018-04-01 14:34 UTC (permalink / raw)
To: netdev; +Cc: davem, jiri, petrm, dsahern, mlxsw, Ido Schimmel
In-Reply-To: <20180401143459.30770-1-idosch@mellanox.com>
From: Jiri Pirko <jiri@mellanox.com>
First arg of these helpers should be "mlxsw_core".
Signed-off-by: Jiri Pirko <jiri@mellanox.com>
Signed-off-by: Ido Schimmel <idosch@mellanox.com>
---
drivers/net/ethernet/mellanox/mlxsw/core.h | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlxsw/core.h b/drivers/net/ethernet/mellanox/mlxsw/core.h
index fd30eaf40475..0d6452699364 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/core.h
+++ b/drivers/net/ethernet/mellanox/mlxsw/core.h
@@ -325,14 +325,14 @@ int mlxsw_core_kvd_sizes_get(struct mlxsw_core *mlxsw_core,
bool mlxsw_core_res_valid(struct mlxsw_core *mlxsw_core,
enum mlxsw_res_id res_id);
-#define MLXSW_CORE_RES_VALID(res, short_res_id) \
- mlxsw_core_res_valid(res, MLXSW_RES_ID_##short_res_id)
+#define MLXSW_CORE_RES_VALID(mlxsw_core, short_res_id) \
+ mlxsw_core_res_valid(mlxsw_core, MLXSW_RES_ID_##short_res_id)
u64 mlxsw_core_res_get(struct mlxsw_core *mlxsw_core,
enum mlxsw_res_id res_id);
-#define MLXSW_CORE_RES_GET(res, short_res_id) \
- mlxsw_core_res_get(res, MLXSW_RES_ID_##short_res_id)
+#define MLXSW_CORE_RES_GET(mlxsw_core, short_res_id) \
+ mlxsw_core_res_get(mlxsw_core, MLXSW_RES_ID_##short_res_id)
#define MLXSW_BUS_F_TXRX BIT(0)
--
2.14.3
^ permalink raw reply related
* [PATCH v2 net-next 07/10] mlxsw: Move "used_kvd_sizes" check to mlxsw_pci_config_profile
From: Ido Schimmel @ 2018-04-01 14:34 UTC (permalink / raw)
To: netdev; +Cc: davem, jiri, petrm, dsahern, mlxsw, Ido Schimmel
In-Reply-To: <20180401143459.30770-1-idosch@mellanox.com>
From: Jiri Pirko <jiri@mellanox.com>
The check should be done directly in mlxsw_pci_config_profile, as for
other profile items. Also, be consistent in naming with the rest and
rename to "used_kvd_sizes".
Signed-off-by: Jiri Pirko <jiri@mellanox.com>
Signed-off-by: Ido Schimmel <idosch@mellanox.com>
---
drivers/net/ethernet/mellanox/mlxsw/core.h | 3 +--
drivers/net/ethernet/mellanox/mlxsw/pci.c | 2 +-
drivers/net/ethernet/mellanox/mlxsw/spectrum.c | 5 ++---
3 files changed, 4 insertions(+), 6 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlxsw/core.h b/drivers/net/ethernet/mellanox/mlxsw/core.h
index 0d6452699364..ff9daa09341d 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/core.h
+++ b/drivers/net/ethernet/mellanox/mlxsw/core.h
@@ -235,8 +235,7 @@ struct mlxsw_config_profile {
used_max_pkey:1,
used_ar_sec:1,
used_adaptive_routing_group_cap:1,
- used_kvd_split_data:1; /* indicate for the kvd's values */
-
+ used_kvd_sizes:1;
u8 max_vepa_channels;
u16 max_mid;
u16 max_pgt;
diff --git a/drivers/net/ethernet/mellanox/mlxsw/pci.c b/drivers/net/ethernet/mellanox/mlxsw/pci.c
index e30c6ce3dcb4..5ab068aec033 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/pci.c
+++ b/drivers/net/ethernet/mellanox/mlxsw/pci.c
@@ -1164,7 +1164,7 @@ static int mlxsw_pci_config_profile(struct mlxsw_pci *mlxsw_pci, char *mbox,
mlxsw_cmd_mbox_config_profile_adaptive_routing_group_cap_set(
mbox, profile->adaptive_routing_group_cap);
}
- if (MLXSW_RES_VALID(res, KVD_SIZE)) {
+ if (profile->used_kvd_sizes && MLXSW_RES_VALID(res, KVD_SIZE)) {
err = mlxsw_pci_profile_get_kvd_sizes(mlxsw_pci, profile, res);
if (err)
return err;
diff --git a/drivers/net/ethernet/mellanox/mlxsw/spectrum.c b/drivers/net/ethernet/mellanox/mlxsw/spectrum.c
index d503cdbeae29..12062aab13c5 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/spectrum.c
+++ b/drivers/net/ethernet/mellanox/mlxsw/spectrum.c
@@ -3793,7 +3793,7 @@ static const struct mlxsw_config_profile mlxsw_sp_config_profile = {
.max_ib_mc = 0,
.used_max_pkey = 1,
.max_pkey = 0,
- .used_kvd_split_data = 1,
+ .used_kvd_sizes = 1,
.kvd_hash_single_parts = 59,
.kvd_hash_double_parts = 41,
.kvd_linear_size = MLXSW_SP_KVD_LINEAR_SIZE,
@@ -3934,8 +3934,7 @@ static int mlxsw_sp_kvd_sizes_get(struct mlxsw_core *mlxsw_core,
int err;
if (!MLXSW_CORE_RES_VALID(mlxsw_core, KVD_SINGLE_MIN_SIZE) ||
- !MLXSW_CORE_RES_VALID(mlxsw_core, KVD_DOUBLE_MIN_SIZE) ||
- !profile->used_kvd_split_data)
+ !MLXSW_CORE_RES_VALID(mlxsw_core, KVD_DOUBLE_MIN_SIZE))
return -EIO;
/* The hash part is what left of the kvd without the
--
2.14.3
^ permalink raw reply related
* [PATCH v2 net-next 08/10] mlxsw: Move "resources_query_enable" out of mlxsw_config_profile
From: Ido Schimmel @ 2018-04-01 14:34 UTC (permalink / raw)
To: netdev; +Cc: davem, jiri, petrm, dsahern, mlxsw, Ido Schimmel
In-Reply-To: <20180401143459.30770-1-idosch@mellanox.com>
From: Jiri Pirko <jiri@mellanox.com>
As struct mlxsw_config_profile is mapped to the payload of the FW
command of the same name, resources_query_enable flag does not belong
there. Move it to struct mlxsw_driver.
Signed-off-by: Jiri Pirko <jiri@mellanox.com>
Signed-off-by: Ido Schimmel <idosch@mellanox.com>
---
drivers/net/ethernet/mellanox/mlxsw/core.c | 5 +++--
drivers/net/ethernet/mellanox/mlxsw/core.h | 2 +-
drivers/net/ethernet/mellanox/mlxsw/pci.c | 9 +++------
drivers/net/ethernet/mellanox/mlxsw/spectrum.c | 2 +-
drivers/net/ethernet/mellanox/mlxsw/switchib.c | 1 -
drivers/net/ethernet/mellanox/mlxsw/switchx2.c | 1 -
6 files changed, 8 insertions(+), 12 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlxsw/core.c b/drivers/net/ethernet/mellanox/mlxsw/core.c
index 3529b545675d..93ea56620a24 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/core.c
+++ b/drivers/net/ethernet/mellanox/mlxsw/core.c
@@ -1008,6 +1008,7 @@ int mlxsw_core_bus_device_register(const struct mlxsw_bus_info *mlxsw_bus_info,
const char *device_kind = mlxsw_bus_info->device_kind;
struct mlxsw_core *mlxsw_core;
struct mlxsw_driver *mlxsw_driver;
+ struct mlxsw_res *res;
size_t alloc_size;
int err;
@@ -1032,8 +1033,8 @@ int mlxsw_core_bus_device_register(const struct mlxsw_bus_info *mlxsw_bus_info,
mlxsw_core->bus_priv = bus_priv;
mlxsw_core->bus_info = mlxsw_bus_info;
- err = mlxsw_bus->init(bus_priv, mlxsw_core, mlxsw_driver->profile,
- &mlxsw_core->res);
+ res = mlxsw_driver->res_query_enabled ? &mlxsw_core->res : NULL;
+ err = mlxsw_bus->init(bus_priv, mlxsw_core, mlxsw_driver->profile, res);
if (err)
goto err_bus_init;
diff --git a/drivers/net/ethernet/mellanox/mlxsw/core.h b/drivers/net/ethernet/mellanox/mlxsw/core.h
index ff9daa09341d..092d39399f3c 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/core.h
+++ b/drivers/net/ethernet/mellanox/mlxsw/core.h
@@ -257,7 +257,6 @@ struct mlxsw_config_profile {
u32 kvd_linear_size;
u8 kvd_hash_single_parts;
u8 kvd_hash_double_parts;
- u8 resource_query_enable;
struct mlxsw_swid_config swid_config[MLXSW_CONFIG_PROFILE_SWID_COUNT];
};
@@ -314,6 +313,7 @@ struct mlxsw_driver {
u64 *p_linear_size);
u8 txhdr_len;
const struct mlxsw_config_profile *profile;
+ bool res_query_enabled;
};
int mlxsw_core_kvd_sizes_get(struct mlxsw_core *mlxsw_core,
diff --git a/drivers/net/ethernet/mellanox/mlxsw/pci.c b/drivers/net/ethernet/mellanox/mlxsw/pci.c
index 5ab068aec033..3a9381977d6d 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/pci.c
+++ b/drivers/net/ethernet/mellanox/mlxsw/pci.c
@@ -1015,16 +1015,14 @@ mlxsw_pci_config_profile_swid_config(struct mlxsw_pci *mlxsw_pci,
}
static int mlxsw_pci_resources_query(struct mlxsw_pci *mlxsw_pci, char *mbox,
- struct mlxsw_res *res,
- u8 query_enabled)
+ struct mlxsw_res *res)
{
int index, i;
u64 data;
u16 id;
int err;
- /* Not all the versions support resources query */
- if (!query_enabled)
+ if (!res)
return 0;
mlxsw_cmd_mbox_zero(mbox);
@@ -1376,8 +1374,7 @@ static int mlxsw_pci_init(void *bus_priv, struct mlxsw_core *mlxsw_core,
if (err)
goto err_boardinfo;
- err = mlxsw_pci_resources_query(mlxsw_pci, mbox, res,
- profile->resource_query_enable);
+ err = mlxsw_pci_resources_query(mlxsw_pci, mbox, res);
if (err)
goto err_query_resources;
diff --git a/drivers/net/ethernet/mellanox/mlxsw/spectrum.c b/drivers/net/ethernet/mellanox/mlxsw/spectrum.c
index 12062aab13c5..b831af38e0a1 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/spectrum.c
+++ b/drivers/net/ethernet/mellanox/mlxsw/spectrum.c
@@ -3803,7 +3803,6 @@ static const struct mlxsw_config_profile mlxsw_sp_config_profile = {
.type = MLXSW_PORT_SWID_TYPE_ETH,
}
},
- .resource_query_enable = 1,
};
static u64 mlxsw_sp_resource_kvd_linear_occ_get(struct devlink *devlink)
@@ -4002,6 +4001,7 @@ static struct mlxsw_driver mlxsw_sp_driver = {
.kvd_sizes_get = mlxsw_sp_kvd_sizes_get,
.txhdr_len = MLXSW_TXHDR_LEN,
.profile = &mlxsw_sp_config_profile,
+ .res_query_enabled = true,
};
bool mlxsw_sp_port_dev_check(const struct net_device *dev)
diff --git a/drivers/net/ethernet/mellanox/mlxsw/switchib.c b/drivers/net/ethernet/mellanox/mlxsw/switchib.c
index ab7a29846bfa..c698ec4fd9d4 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/switchib.c
+++ b/drivers/net/ethernet/mellanox/mlxsw/switchib.c
@@ -510,7 +510,6 @@ static const struct mlxsw_config_profile mlxsw_sib_config_profile = {
.type = MLXSW_PORT_SWID_TYPE_IB,
}
},
- .resource_query_enable = 0,
};
static struct mlxsw_driver mlxsw_sib_driver = {
diff --git a/drivers/net/ethernet/mellanox/mlxsw/switchx2.c b/drivers/net/ethernet/mellanox/mlxsw/switchx2.c
index c87b0934a405..a655c5850aa6 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/switchx2.c
+++ b/drivers/net/ethernet/mellanox/mlxsw/switchx2.c
@@ -1706,7 +1706,6 @@ static const struct mlxsw_config_profile mlxsw_sx_config_profile = {
.type = MLXSW_PORT_SWID_TYPE_IB,
}
},
- .resource_query_enable = 0,
};
static struct mlxsw_driver mlxsw_sx_driver = {
--
2.14.3
^ permalink raw reply related
* [PATCH v2 net-next 09/10] mlxsw: spectrum: Pass mlxsw_core as arg of mlxsw_sp_kvdl_resources_register()
From: Ido Schimmel @ 2018-04-01 14:34 UTC (permalink / raw)
To: netdev; +Cc: davem, jiri, petrm, dsahern, mlxsw, Ido Schimmel
In-Reply-To: <20180401143459.30770-1-idosch@mellanox.com>
From: Jiri Pirko <jiri@mellanox.com>
Pass struct mlxsw_core instead of devlink since it is nicer within mlxsw
code and we need both structs in mlxsw_sp_kvdl_resources_register()
anyway.
Signed-off-by: Jiri Pirko <jiri@mellanox.com>
Signed-off-by: Ido Schimmel <idosch@mellanox.com>
---
drivers/net/ethernet/mellanox/mlxsw/spectrum.c | 2 +-
drivers/net/ethernet/mellanox/mlxsw/spectrum.h | 2 +-
drivers/net/ethernet/mellanox/mlxsw/spectrum_kvdl.c | 4 ++--
3 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlxsw/spectrum.c b/drivers/net/ethernet/mellanox/mlxsw/spectrum.c
index b831af38e0a1..53fffd09d133 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/spectrum.c
+++ b/drivers/net/ethernet/mellanox/mlxsw/spectrum.c
@@ -3892,7 +3892,7 @@ static int mlxsw_sp_resources_register(struct mlxsw_core *mlxsw_core)
if (err)
return err;
- err = mlxsw_sp_kvdl_resources_register(devlink);
+ err = mlxsw_sp_kvdl_resources_register(mlxsw_core);
if (err)
return err;
diff --git a/drivers/net/ethernet/mellanox/mlxsw/spectrum.h b/drivers/net/ethernet/mellanox/mlxsw/spectrum.h
index 21bee8f19894..c4d47e456737 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/spectrum.h
+++ b/drivers/net/ethernet/mellanox/mlxsw/spectrum.h
@@ -443,7 +443,7 @@ int mlxsw_sp_kvdl_alloc_size_query(struct mlxsw_sp *mlxsw_sp,
unsigned int entry_count,
unsigned int *p_alloc_size);
u64 mlxsw_sp_kvdl_occ_get(const struct mlxsw_sp *mlxsw_sp);
-int mlxsw_sp_kvdl_resources_register(struct devlink *devlink);
+int mlxsw_sp_kvdl_resources_register(struct mlxsw_core *mlxsw_core);
struct mlxsw_sp_acl_rule_info {
unsigned int priority;
diff --git a/drivers/net/ethernet/mellanox/mlxsw/spectrum_kvdl.c b/drivers/net/ethernet/mellanox/mlxsw/spectrum_kvdl.c
index 7b28f65d6407..8796db44dcc3 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/spectrum_kvdl.c
+++ b/drivers/net/ethernet/mellanox/mlxsw/spectrum_kvdl.c
@@ -368,9 +368,9 @@ static const struct devlink_resource_ops mlxsw_sp_kvdl_chunks_large_ops = {
.occ_get = mlxsw_sp_kvdl_large_chunks_occ_get,
};
-int mlxsw_sp_kvdl_resources_register(struct devlink *devlink)
+int mlxsw_sp_kvdl_resources_register(struct mlxsw_core *mlxsw_core)
{
- struct mlxsw_core *mlxsw_core = devlink_priv(devlink);
+ struct devlink *devlink = priv_to_devlink(mlxsw_core);
static struct devlink_resource_size_params size_params;
u32 kvdl_max_size;
int err;
--
2.14.3
^ permalink raw reply related
* [PATCH v2 net-next 10/10] mlxsw: spectrum: Don't use resource ID of 0
From: Ido Schimmel @ 2018-04-01 14:34 UTC (permalink / raw)
To: netdev; +Cc: davem, jiri, petrm, dsahern, mlxsw, Ido Schimmel
In-Reply-To: <20180401143459.30770-1-idosch@mellanox.com>
From: Petr Machata <petrm@mellanox.com>
In commit 145307460ba9 ("devlink: Remove top_hierarchy arg to
devlink_resource_register"), the "top_hierarchy" parameter to
devlink_resource_register() was removed in favor of using the parameter
"parent_resource_id" exclusively to determine who the parent is. The
root node's resource ID for this purpose is
DEVLINK_RESOURCE_ID_PARENT_TOP with the value 0. It is therefore
problematic that the resource MLXSW_SP_RESOURCE_KVD has also ID of 0.
Fix this by numbering driver-specific resources from 1.
Fixes: 145307460ba9 ("devlink: Remove top_hierarchy arg to devlink_resource_register")
Signed-off-by: Petr Machata <petrm@mellanox.com>
Signed-off-by: Ido Schimmel <idosch@mellanox.com>
---
drivers/net/ethernet/mellanox/mlxsw/spectrum.h | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/net/ethernet/mellanox/mlxsw/spectrum.h b/drivers/net/ethernet/mellanox/mlxsw/spectrum.h
index c4d47e456737..82820ba43728 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/spectrum.h
+++ b/drivers/net/ethernet/mellanox/mlxsw/spectrum.h
@@ -75,7 +75,7 @@
#define MLXSW_SP_RESOURCE_NAME_KVD_LINEAR_LARGE_CHUNKS "large_chunks"
enum mlxsw_sp_resource_id {
- MLXSW_SP_RESOURCE_KVD,
+ MLXSW_SP_RESOURCE_KVD = 1,
MLXSW_SP_RESOURCE_KVD_LINEAR,
MLXSW_SP_RESOURCE_KVD_HASH_SINGLE,
MLXSW_SP_RESOURCE_KVD_HASH_DOUBLE,
--
2.14.3
^ permalink raw reply related
* [PATCH net] route: check sysctl_fib_multipath_use_neigh earlier than hash
From: Xin Long @ 2018-04-01 14:40 UTC (permalink / raw)
To: network dev; +Cc: davem, Julian Anastasov, David Ahern
Prior to this patch, when one packet is hashed into path [1]
(hash <= nh_upper_bound) and it's neigh is dead, it will try
path [2]. However, if path [2]'s neigh is alive but it's
hash > nh_upper_bound, it will not return this alive path.
This packet will never be sent even if path [2] is alive.
3.3.3.1/24:
nexthop via 1.1.1.254 dev eth1 weight 1 <--[1] (dead neigh)
nexthop via 2.2.2.254 dev eth2 weight 1 <--[2]
With sysctl_fib_multipath_use_neigh set is supposed to find an
available path respecting to the l3/l4 hash. But if there is
no available route with this hash, it should at least return
an alive route even with other hash.
This patch is to fix it by processing fib_multipath_use_neigh
earlier than the hash check, so that it will at least return
an alive route if there is when fib_multipath_use_neigh is
enabled. It's also compatible with before when there are alive
routes with the l3/l4 hash.
Fixes: a6db4494d218 ("net: ipv4: Consider failed nexthops in multipath routes")
Reported-by: Jianlin Shi <jishi@redhat.com>
Signed-off-by: Xin Long <lucien.xin@gmail.com>
---
net/ipv4/fib_semantics.c | 20 +++++++++++---------
1 file changed, 11 insertions(+), 9 deletions(-)
diff --git a/net/ipv4/fib_semantics.c b/net/ipv4/fib_semantics.c
index 7d36a95..9d51292 100644
--- a/net/ipv4/fib_semantics.c
+++ b/net/ipv4/fib_semantics.c
@@ -1746,18 +1746,20 @@ void fib_select_multipath(struct fib_result *res, int hash)
bool first = false;
for_nexthops(fi) {
+ if (net->ipv4.sysctl_fib_multipath_use_neigh) {
+ if (!fib_good_nh(nh))
+ continue;
+ if (!first) {
+ res->nh_sel = nhsel;
+ first = true;
+ }
+ }
+
if (hash > atomic_read(&nh->nh_upper_bound))
continue;
- if (!net->ipv4.sysctl_fib_multipath_use_neigh ||
- fib_good_nh(nh)) {
- res->nh_sel = nhsel;
- return;
- }
- if (!first) {
- res->nh_sel = nhsel;
- first = true;
- }
+ res->nh_sel = nhsel;
+ return;
} endfor_nexthops(fi);
}
#endif
--
2.1.0
^ permalink raw reply related
* [bpf-next PATCH 0/4] Sockmap Updates
From: John Fastabend @ 2018-04-01 15:00 UTC (permalink / raw)
To: ast, daniel; +Cc: netdev, davem
Some additions for sockmap. First a couple fixes from the last round
to ensure (a) we account for corked memory when a socket is closing
and (b) do not set a null sk_prot value when tcp_close and sockmap
destroy race.
Then the new addition is a hash map type. As the sockmap use cases
become more complex and the number of socks involved becomes larger
mapping and tracking the socks in the array complicates the control
plane quite a bit. By adding a hash mode we are able to simplify this.
A couple design decisions were made here. First I did not try to
embed this in the existing hashtab logic. If we had done this
we lose the ability to build custom htab and element data structures.
This ended up complicating both hashtab and sockmap. This was the
same rational for making both sockmap and devmap distinct objects
outside of the standard array map. The slight downside of this
decision is some code becomes duplicated. The biggest offender is
get_next_key, which is close to a cut'n'paste. However, this
seems better than forcing sockmap.c to follow hashtab data
structures limiting our opportunities for optimizations.
The second decision was to use a new set of helper routines that
are distinct to the new SOCKHASH map type. Because the original
sockmap API (sk_* helpers) used a u32 (and is already in UAPI) it
could not be changed to a 'void *key' type needed for SOCKHASH. The
msg hooks are not in a released UAPI but we use sk_* helpers and
msg_* helpers together in our programs and creating an API with
different patterns depending on context seemed undesireable. So
this series adds two sets of helpers one for SOCKMAP and one for
SOCKHASH. The nice fallout of this is the verifier knows these
signatures and is happy to throw errors when users use the wrong
types.
For testing I changed the sample sockmap program to use SOCKHASH
maps and verified everything passed as expected with a couple
different key sizes.
The test patches will be added after sockmap sample is converted
into selftests. This work is currently in-progress because the
selftests API needs to be extended to support multiple programs
in as single file with different types.
---
John Fastabend (4):
bpf: sockmap, free memory on sock close with cork data
bpf: sockmap, duplicates release calls may NULL sk_prot
bpf: sockmap, refactor sockmap routines to work with hashmap
bpf: sockmap, add hash map support
include/linux/bpf.h | 8
include/linux/bpf_types.h | 1
include/linux/filter.h | 3
include/net/tcp.h | 3
include/uapi/linux/bpf.h | 6
kernel/bpf/core.c | 1
kernel/bpf/sockmap.c | 667 ++++++++++++++++++++++++++---
kernel/bpf/verifier.c | 14 +
net/core/filter.c | 81 +++-
tools/bpf/bpftool/map.c | 1
tools/include/uapi/linux/bpf.h | 6
tools/testing/selftests/bpf/bpf_helpers.h | 13 -
12 files changed, 700 insertions(+), 104 deletions(-)
^ permalink raw reply
* [bpf-next PATCH 1/4] bpf: sockmap, free memory on sock close with cork data
From: John Fastabend @ 2018-04-01 15:00 UTC (permalink / raw)
To: ast, daniel; +Cc: netdev, davem
In-Reply-To: <20180401145728.24727.706.stgit@john-Precision-Tower-5810>
If a socket with pending cork data is closed we do not return the
memory to the socket until the garbage collector free's the psock
structure. The garbage collector though can run after the sock has
completed its close operation. If this ordering happens the sock code
will through a WARN_ON because there is still outstanding memory
accounted to the sock.
To resolve this ensure we return memory to the sock when a socket
is closed.
Signed-off-by: John Fastabend <john.fastabend@gmail.com>
Fixes: 91843d540a13 ("bpf: sockmap, add msg_cork_bytes() helper")
---
kernel/bpf/sockmap.c | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/kernel/bpf/sockmap.c b/kernel/bpf/sockmap.c
index d2bda5a..8ddf326 100644
--- a/kernel/bpf/sockmap.c
+++ b/kernel/bpf/sockmap.c
@@ -211,6 +211,12 @@ static void bpf_tcp_close(struct sock *sk, long timeout)
close_fun = psock->save_close;
write_lock_bh(&sk->sk_callback_lock);
+ if (psock->cork) {
+ free_start_sg(psock->sock, psock->cork);
+ kfree(psock->cork);
+ psock->cork = NULL;
+ }
+
list_for_each_entry_safe(md, mtmp, &psock->ingress, list) {
list_del(&md->list);
free_start_sg(psock->sock, md);
^ permalink raw reply related
* [bpf-next PATCH 2/4] bpf: sockmap, duplicates release calls may NULL sk_prot
From: John Fastabend @ 2018-04-01 15:00 UTC (permalink / raw)
To: ast, daniel; +Cc: netdev, davem
In-Reply-To: <20180401145728.24727.706.stgit@john-Precision-Tower-5810>
It is possible to have multiple ULP tcp_release call paths in flight
if a sock is closed and simultaneously being removed from the sockmap
control path. The result would be setting the sk_prot to the saved
values on the first iteration and then on the second iteration setting
the value to NULL.
This patch resolves this by ensuring we only reset the sk_prot pointer
if we have a valid saved state to set.
Fixes: 4f738adba30a7 ("bpf: create tcp_bpf_ulp allowing BPF to monitor socket TX/RX data")
Signed-off-by: John Fastabend <john.fastabend@gmail.com>
---
kernel/bpf/sockmap.c | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/kernel/bpf/sockmap.c b/kernel/bpf/sockmap.c
index 8ddf326..8dd9210 100644
--- a/kernel/bpf/sockmap.c
+++ b/kernel/bpf/sockmap.c
@@ -182,8 +182,10 @@ static void bpf_tcp_release(struct sock *sk)
psock->cork = NULL;
}
- sk->sk_prot = psock->sk_proto;
- psock->sk_proto = NULL;
+ if (psock->sk_proto) {
+ sk->sk_prot = psock->sk_proto;
+ psock->sk_proto = NULL;
+ }
out:
rcu_read_unlock();
}
^ permalink raw reply related
* [bpf-next PATCH 3/4] bpf: sockmap, refactor sockmap routines to work with hashmap
From: John Fastabend @ 2018-04-01 15:01 UTC (permalink / raw)
To: ast, daniel; +Cc: netdev, davem
In-Reply-To: <20180401145728.24727.706.stgit@john-Precision-Tower-5810>
This patch only refactors the existing sockmap code. This will allow
much of the psock initialization code path and bpf helper codes to
work for both sockmap bpf map types that are backed by an array, the
currently supported type, and the new hash backed bpf map type
sockhash.
Most the fallout comes from three changes,
- Pushing bpf programs into an independent structure so we
can use it from the htab struct in the next patch.
- Generalizing helpers to use void *key instead of the hardcoded
u32.
- Instead of passing map/key through the metadata we now do
the lookup inline. This avoids storing the key in the metadata
which will be useful when keys can be longer than 4 bytes.
Signed-off-by: John Fastabend <john.fastabend@gmail.com>
---
include/linux/filter.h | 3 -
include/net/tcp.h | 3 -
kernel/bpf/sockmap.c | 149 +++++++++++++++++++++++++++++-------------------
net/core/filter.c | 27 +--------
4 files changed, 95 insertions(+), 87 deletions(-)
diff --git a/include/linux/filter.h b/include/linux/filter.h
index fc4e8f9..1d47160 100644
--- a/include/linux/filter.h
+++ b/include/linux/filter.h
@@ -519,9 +519,8 @@ struct sk_msg_buff {
int sg_end;
struct scatterlist sg_data[MAX_SKB_FRAGS];
bool sg_copy[MAX_SKB_FRAGS];
- __u32 key;
__u32 flags;
- struct bpf_map *map;
+ struct sock *sk;
struct sk_buff *skb;
struct list_head list;
};
diff --git a/include/net/tcp.h b/include/net/tcp.h
index 9c9b376..c3124d9 100644
--- a/include/net/tcp.h
+++ b/include/net/tcp.h
@@ -810,9 +810,8 @@ struct tcp_skb_cb {
#endif
} header; /* For incoming skbs */
struct {
- __u32 key;
__u32 flags;
- struct bpf_map *map;
+ struct sock *sk;
void *data_end;
} bpf;
};
diff --git a/kernel/bpf/sockmap.c b/kernel/bpf/sockmap.c
index 8dd9210..c006e3b 100644
--- a/kernel/bpf/sockmap.c
+++ b/kernel/bpf/sockmap.c
@@ -47,14 +47,18 @@
#define SOCK_CREATE_FLAG_MASK \
(BPF_F_NUMA_NODE | BPF_F_RDONLY | BPF_F_WRONLY)
-struct bpf_stab {
- struct bpf_map map;
- struct sock **sock_map;
+struct bpf_sock_progs {
struct bpf_prog *bpf_tx_msg;
struct bpf_prog *bpf_parse;
struct bpf_prog *bpf_verdict;
};
+struct bpf_stab {
+ struct bpf_map map;
+ struct sock **sock_map;
+ struct bpf_sock_progs progs;
+};
+
enum smap_psock_state {
SMAP_TX_RUNNING,
};
@@ -455,7 +459,7 @@ static int free_curr_sg(struct sock *sk, struct sk_msg_buff *md)
static int bpf_map_msg_verdict(int _rc, struct sk_msg_buff *md)
{
return ((_rc == SK_PASS) ?
- (md->map ? __SK_REDIRECT : __SK_PASS) :
+ (md->sk ? __SK_REDIRECT : __SK_PASS) :
__SK_DROP);
}
@@ -1045,7 +1049,7 @@ static int smap_verdict_func(struct smap_psock *psock, struct sk_buff *skb)
* when we orphan the skb so that we don't have the possibility
* to reference a stale map.
*/
- TCP_SKB_CB(skb)->bpf.map = NULL;
+ TCP_SKB_CB(skb)->bpf.sk = NULL;
skb->sk = psock->sock;
bpf_compute_data_pointers(skb);
preempt_disable();
@@ -1055,7 +1059,7 @@ static int smap_verdict_func(struct smap_psock *psock, struct sk_buff *skb)
/* Moving return codes from UAPI namespace into internal namespace */
return rc == SK_PASS ?
- (TCP_SKB_CB(skb)->bpf.map ? __SK_REDIRECT : __SK_PASS) :
+ (TCP_SKB_CB(skb)->bpf.sk ? __SK_REDIRECT : __SK_PASS) :
__SK_DROP;
}
@@ -1325,7 +1329,6 @@ static int smap_init_sock(struct smap_psock *psock,
}
static void smap_init_progs(struct smap_psock *psock,
- struct bpf_stab *stab,
struct bpf_prog *verdict,
struct bpf_prog *parse)
{
@@ -1403,14 +1406,13 @@ static void smap_gc_work(struct work_struct *w)
kfree(psock);
}
-static struct smap_psock *smap_init_psock(struct sock *sock,
- struct bpf_stab *stab)
+static struct smap_psock *smap_init_psock(struct sock *sock, int node)
{
struct smap_psock *psock;
psock = kzalloc_node(sizeof(struct smap_psock),
GFP_ATOMIC | __GFP_NOWARN,
- stab->map.numa_node);
+ node);
if (!psock)
return ERR_PTR(-ENOMEM);
@@ -1618,40 +1620,27 @@ static int sock_map_delete_elem(struct bpf_map *map, void *key)
* - sock_map must use READ_ONCE and (cmp)xchg operations
* - BPF verdict/parse programs must use READ_ONCE and xchg operations
*/
-static int sock_map_ctx_update_elem(struct bpf_sock_ops_kern *skops,
- struct bpf_map *map,
- void *key, u64 flags)
+
+static int __sock_map_ctx_update_elem(struct bpf_map *map,
+ struct bpf_sock_progs *progs,
+ struct sock *sock,
+ struct sock **map_link,
+ struct sock **hash_link,
+ void *key)
{
- struct bpf_stab *stab = container_of(map, struct bpf_stab, map);
- struct smap_psock_map_entry *e = NULL;
struct bpf_prog *verdict, *parse, *tx_msg;
- struct sock *osock, *sock;
+ struct smap_psock_map_entry *e = NULL;
struct smap_psock *psock;
- u32 i = *(u32 *)key;
bool new = false;
int err;
- if (unlikely(flags > BPF_EXIST))
- return -EINVAL;
-
- if (unlikely(i >= stab->map.max_entries))
- return -E2BIG;
-
- sock = READ_ONCE(stab->sock_map[i]);
- if (flags == BPF_EXIST && !sock)
- return -ENOENT;
- else if (flags == BPF_NOEXIST && sock)
- return -EEXIST;
-
- sock = skops->sk;
-
/* 1. If sock map has BPF programs those will be inherited by the
* sock being added. If the sock is already attached to BPF programs
* this results in an error.
*/
- verdict = READ_ONCE(stab->bpf_verdict);
- parse = READ_ONCE(stab->bpf_parse);
- tx_msg = READ_ONCE(stab->bpf_tx_msg);
+ verdict = READ_ONCE(progs->bpf_verdict);
+ parse = READ_ONCE(progs->bpf_parse);
+ tx_msg = READ_ONCE(progs->bpf_tx_msg);
if (parse && verdict) {
/* bpf prog refcnt may be zero if a concurrent attach operation
@@ -1659,11 +1648,11 @@ static int sock_map_ctx_update_elem(struct bpf_sock_ops_kern *skops,
* we increment the refcnt. If this is the case abort with an
* error.
*/
- verdict = bpf_prog_inc_not_zero(stab->bpf_verdict);
+ verdict = bpf_prog_inc_not_zero(progs->bpf_verdict);
if (IS_ERR(verdict))
return PTR_ERR(verdict);
- parse = bpf_prog_inc_not_zero(stab->bpf_parse);
+ parse = bpf_prog_inc_not_zero(progs->bpf_parse);
if (IS_ERR(parse)) {
bpf_prog_put(verdict);
return PTR_ERR(parse);
@@ -1671,7 +1660,7 @@ static int sock_map_ctx_update_elem(struct bpf_sock_ops_kern *skops,
}
if (tx_msg) {
- tx_msg = bpf_prog_inc_not_zero(stab->bpf_tx_msg);
+ tx_msg = bpf_prog_inc_not_zero(progs->bpf_tx_msg);
if (IS_ERR(tx_msg)) {
if (verdict)
bpf_prog_put(verdict);
@@ -1704,7 +1693,7 @@ static int sock_map_ctx_update_elem(struct bpf_sock_ops_kern *skops,
goto out_progs;
}
} else {
- psock = smap_init_psock(sock, stab);
+ psock = smap_init_psock(sock, map->numa_node);
if (IS_ERR(psock)) {
err = PTR_ERR(psock);
goto out_progs;
@@ -1719,7 +1708,6 @@ static int sock_map_ctx_update_elem(struct bpf_sock_ops_kern *skops,
err = -ENOMEM;
goto out_progs;
}
- e->entry = &stab->sock_map[i];
/* 3. At this point we have a reference to a valid psock that is
* running. Attach any BPF programs needed.
@@ -1736,7 +1724,7 @@ static int sock_map_ctx_update_elem(struct bpf_sock_ops_kern *skops,
err = smap_init_sock(psock, sock);
if (err)
goto out_free;
- smap_init_progs(psock, stab, verdict, parse);
+ smap_init_progs(psock, verdict, parse);
smap_start_sock(psock, sock);
}
@@ -1745,19 +1733,12 @@ static int sock_map_ctx_update_elem(struct bpf_sock_ops_kern *skops,
* it with. Because we can only have a single set of programs if
* old_sock has a strp we can stop it.
*/
- list_add_tail(&e->list, &psock->maps);
- write_unlock_bh(&sock->sk_callback_lock);
-
- osock = xchg(&stab->sock_map[i], sock);
- if (osock) {
- struct smap_psock *opsock = smap_psock_sk(osock);
-
- write_lock_bh(&osock->sk_callback_lock);
- smap_list_remove(opsock, &stab->sock_map[i]);
- smap_release_sock(opsock, osock);
- write_unlock_bh(&osock->sk_callback_lock);
+ if (map_link) {
+ e->entry = map_link;
+ list_add_tail(&e->list, &psock->maps);
}
- return 0;
+ write_unlock_bh(&sock->sk_callback_lock);
+ return err;
out_free:
smap_release_sock(psock, sock);
out_progs:
@@ -1772,23 +1753,69 @@ static int sock_map_ctx_update_elem(struct bpf_sock_ops_kern *skops,
return err;
}
-int sock_map_prog(struct bpf_map *map, struct bpf_prog *prog, u32 type)
+static int sock_map_ctx_update_elem(struct bpf_sock_ops_kern *skops,
+ struct bpf_map *map,
+ void *key, u64 flags)
{
struct bpf_stab *stab = container_of(map, struct bpf_stab, map);
+ struct bpf_sock_progs *progs = &stab->progs;
+ struct sock *osock, *sock;
+ u32 i = *(u32 *)key;
+ int err;
+
+ if (unlikely(flags > BPF_EXIST))
+ return -EINVAL;
+
+ if (unlikely(i >= stab->map.max_entries))
+ return -E2BIG;
+
+ sock = READ_ONCE(stab->sock_map[i]);
+ if (flags == BPF_EXIST && !sock)
+ return -ENOENT;
+ else if (flags == BPF_NOEXIST && sock)
+ return -EEXIST;
+
+ sock = skops->sk;
+ err = __sock_map_ctx_update_elem(map, progs, sock, &stab->sock_map[i],
+ NULL, key);
+ if (err)
+ goto out;
+
+ osock = xchg(&stab->sock_map[i], sock);
+ if (osock) {
+ struct smap_psock *opsock = smap_psock_sk(osock);
+
+ write_lock_bh(&osock->sk_callback_lock);
+ smap_list_remove(opsock, &stab->sock_map[i]);
+ smap_release_sock(opsock, osock);
+ write_unlock_bh(&osock->sk_callback_lock);
+ }
+out:
+ return 0;
+}
+
+int sock_map_prog(struct bpf_map *map, struct bpf_prog *prog, u32 type)
+{
+ struct bpf_sock_progs *progs;
struct bpf_prog *orig;
- if (unlikely(map->map_type != BPF_MAP_TYPE_SOCKMAP))
+ if (map->map_type == BPF_MAP_TYPE_SOCKMAP) {
+ struct bpf_stab *stab = container_of(map, struct bpf_stab, map);
+
+ progs = &stab->progs;
+ } else {
return -EINVAL;
+ }
switch (type) {
case BPF_SK_MSG_VERDICT:
- orig = xchg(&stab->bpf_tx_msg, prog);
+ orig = xchg(&progs->bpf_tx_msg, prog);
break;
case BPF_SK_SKB_STREAM_PARSER:
- orig = xchg(&stab->bpf_parse, prog);
+ orig = xchg(&progs->bpf_parse, prog);
break;
case BPF_SK_SKB_STREAM_VERDICT:
- orig = xchg(&stab->bpf_verdict, prog);
+ orig = xchg(&progs->bpf_verdict, prog);
break;
default:
return -EOPNOTSUPP;
@@ -1837,16 +1864,18 @@ static int sock_map_update_elem(struct bpf_map *map,
static void sock_map_release(struct bpf_map *map, struct file *map_file)
{
struct bpf_stab *stab = container_of(map, struct bpf_stab, map);
+ struct bpf_sock_progs *progs;
struct bpf_prog *orig;
- orig = xchg(&stab->bpf_parse, NULL);
+ progs = &stab->progs;
+ orig = xchg(&progs->bpf_parse, NULL);
if (orig)
bpf_prog_put(orig);
- orig = xchg(&stab->bpf_verdict, NULL);
+ orig = xchg(&progs->bpf_verdict, NULL);
if (orig)
bpf_prog_put(orig);
- orig = xchg(&stab->bpf_tx_msg, NULL);
+ orig = xchg(&progs->bpf_tx_msg, NULL);
if (orig)
bpf_prog_put(orig);
}
diff --git a/net/core/filter.c b/net/core/filter.c
index d31aff9..6456e9d 100644
--- a/net/core/filter.c
+++ b/net/core/filter.c
@@ -1859,9 +1859,8 @@ int skb_do_redirect(struct sk_buff *skb)
if (unlikely(flags & ~(BPF_F_INGRESS)))
return SK_DROP;
- tcb->bpf.key = key;
tcb->bpf.flags = flags;
- tcb->bpf.map = map;
+ tcb->bpf.sk = __sock_map_lookup_elem(map, key);
return SK_PASS;
}
@@ -1869,16 +1868,8 @@ int skb_do_redirect(struct sk_buff *skb)
struct sock *do_sk_redirect_map(struct sk_buff *skb)
{
struct tcp_skb_cb *tcb = TCP_SKB_CB(skb);
- struct sock *sk = NULL;
- if (tcb->bpf.map) {
- sk = __sock_map_lookup_elem(tcb->bpf.map, tcb->bpf.key);
-
- tcb->bpf.key = 0;
- tcb->bpf.map = NULL;
- }
-
- return sk;
+ return tcb->bpf.sk;
}
static const struct bpf_func_proto bpf_sk_redirect_map_proto = {
@@ -1898,25 +1889,15 @@ struct sock *do_sk_redirect_map(struct sk_buff *skb)
if (unlikely(flags & ~(BPF_F_INGRESS)))
return SK_DROP;
- msg->key = key;
msg->flags = flags;
- msg->map = map;
+ msg->sk = __sock_map_lookup_elem(map, key);
return SK_PASS;
}
struct sock *do_msg_redirect_map(struct sk_msg_buff *msg)
{
- struct sock *sk = NULL;
-
- if (msg->map) {
- sk = __sock_map_lookup_elem(msg->map, msg->key);
-
- msg->key = 0;
- msg->map = NULL;
- }
-
- return sk;
+ return msg->sk;
}
static const struct bpf_func_proto bpf_msg_redirect_map_proto = {
^ permalink raw reply related
* [bpf-next PATCH 4/4] bpf: sockmap, add hash map support
From: John Fastabend @ 2018-04-01 15:01 UTC (permalink / raw)
To: ast, daniel; +Cc: netdev, davem
In-Reply-To: <20180401145728.24727.706.stgit@john-Precision-Tower-5810>
Sockmap is currently backed by an array and enforces keys to be
four bytes. This works well for many use cases and was originally
modeled after devmap which also uses four bytes keys. However,
this has become limiting in larger use cases where a hash would
be more appropriate. For example users may want to use the 5-tuple
of the socket as the lookup key.
To support this add hash support.
Signed-off-by: John Fastabend <john.fastabend@gmail.com>
---
include/linux/bpf.h | 8
include/linux/bpf_types.h | 1
include/uapi/linux/bpf.h | 6
kernel/bpf/core.c | 1
kernel/bpf/sockmap.c | 512 ++++++++++++++++++++++++++++-
kernel/bpf/verifier.c | 14 +
net/core/filter.c | 54 +++
tools/bpf/bpftool/map.c | 1
tools/include/uapi/linux/bpf.h | 6
tools/testing/selftests/bpf/bpf_helpers.h | 13 +
10 files changed, 598 insertions(+), 18 deletions(-)
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 95a7abd..d5c77f2 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -645,6 +645,7 @@ static inline void bpf_map_offload_map_free(struct bpf_map *map)
#if defined(CONFIG_STREAM_PARSER) && defined(CONFIG_BPF_SYSCALL) && defined(CONFIG_INET)
struct sock *__sock_map_lookup_elem(struct bpf_map *map, u32 key);
+struct sock *__sock_hash_lookup_elem(struct bpf_map *map, void *key);
int sock_map_prog(struct bpf_map *map, struct bpf_prog *prog, u32 type);
#else
static inline struct sock *__sock_map_lookup_elem(struct bpf_map *map, u32 key)
@@ -652,6 +653,12 @@ static inline struct sock *__sock_map_lookup_elem(struct bpf_map *map, u32 key)
return NULL;
}
+static inline struct sock *__sock_hash_lookup_elem(struct bpf_map *map,
+ void *key)
+{
+ return NULL;
+}
+
static inline int sock_map_prog(struct bpf_map *map,
struct bpf_prog *prog,
u32 type)
@@ -677,6 +684,7 @@ static inline int sock_map_prog(struct bpf_map *map,
extern const struct bpf_func_proto bpf_skb_vlan_pop_proto;
extern const struct bpf_func_proto bpf_get_stackid_proto;
extern const struct bpf_func_proto bpf_sock_map_update_proto;
+extern const struct bpf_func_proto bpf_sock_hash_update_proto;
/* Shared helpers among cBPF and eBPF. */
void bpf_user_rnd_init_once(void);
diff --git a/include/linux/bpf_types.h b/include/linux/bpf_types.h
index 2b28fcf..3101118 100644
--- a/include/linux/bpf_types.h
+++ b/include/linux/bpf_types.h
@@ -47,6 +47,7 @@
BPF_MAP_TYPE(BPF_MAP_TYPE_DEVMAP, dev_map_ops)
#if defined(CONFIG_STREAM_PARSER) && defined(CONFIG_INET)
BPF_MAP_TYPE(BPF_MAP_TYPE_SOCKMAP, sock_map_ops)
+BPF_MAP_TYPE(BPF_MAP_TYPE_SOCKHASH, sock_hash_ops)
#endif
BPF_MAP_TYPE(BPF_MAP_TYPE_CPUMAP, cpu_map_ops)
#endif
diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h
index c5ec897..9886e88 100644
--- a/include/uapi/linux/bpf.h
+++ b/include/uapi/linux/bpf.h
@@ -115,6 +115,7 @@ enum bpf_map_type {
BPF_MAP_TYPE_DEVMAP,
BPF_MAP_TYPE_SOCKMAP,
BPF_MAP_TYPE_CPUMAP,
+ BPF_MAP_TYPE_SOCKHASH,
};
enum bpf_prog_type {
@@ -821,7 +822,10 @@ struct bpf_stack_build_id {
FN(msg_apply_bytes), \
FN(msg_cork_bytes), \
FN(msg_pull_data), \
- FN(bind),
+ FN(bind), \
+ FN(sock_hash_update), \
+ FN(msg_redirect_hash), \
+ FN(sk_redirect_hash),
/* integer value in 'imm' field of BPF_CALL instruction selects which helper
* function eBPF program intends to call
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index d315b39..dd2c906 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -1769,6 +1769,7 @@ void bpf_user_rnd_init_once(void)
const struct bpf_func_proto bpf_get_current_uid_gid_proto __weak;
const struct bpf_func_proto bpf_get_current_comm_proto __weak;
const struct bpf_func_proto bpf_sock_map_update_proto __weak;
+const struct bpf_func_proto bpf_sock_hash_update_proto __weak;
const struct bpf_func_proto * __weak bpf_get_trace_printk_proto(void)
{
diff --git a/kernel/bpf/sockmap.c b/kernel/bpf/sockmap.c
index c006e3b..175fcb0 100644
--- a/kernel/bpf/sockmap.c
+++ b/kernel/bpf/sockmap.c
@@ -59,6 +59,28 @@ struct bpf_stab {
struct bpf_sock_progs progs;
};
+struct bucket {
+ struct hlist_nulls_head head;
+ raw_spinlock_t lock;
+};
+
+struct bpf_htab {
+ struct bpf_map map;
+ struct bucket *buckets;
+ atomic_t count;
+ u32 n_buckets;
+ u32 elem_size;
+ struct bpf_sock_progs progs;
+};
+
+struct htab_elem {
+ struct rcu_head rcu;
+ struct hlist_nulls_node hash_node;
+ u32 hash;
+ struct sock *sk;
+ char key[0];
+};
+
enum smap_psock_state {
SMAP_TX_RUNNING,
};
@@ -66,6 +88,8 @@ enum smap_psock_state {
struct smap_psock_map_entry {
struct list_head list;
struct sock **entry;
+ struct htab_elem *hash_link;
+ struct bpf_htab *htab;
};
struct smap_psock {
@@ -194,6 +218,27 @@ static void bpf_tcp_release(struct sock *sk)
rcu_read_unlock();
}
+static void htab_elem_free_rcu(struct rcu_head *head)
+{
+ struct htab_elem *l = container_of(head, struct htab_elem, rcu);
+
+ /* must increment bpf_prog_active to avoid kprobe+bpf triggering while
+ * we're calling kfree, otherwise deadlock is possible if kprobes
+ * are placed somewhere inside of slub
+ */
+ preempt_disable();
+ __this_cpu_inc(bpf_prog_active);
+ kfree(l);
+ __this_cpu_dec(bpf_prog_active);
+ preempt_enable();
+}
+
+static void free_htab_elem(struct bpf_htab *htab, struct htab_elem *l)
+{
+ atomic_dec(&htab->count);
+ call_rcu(&l->rcu, htab_elem_free_rcu);
+}
+
static void bpf_tcp_close(struct sock *sk, long timeout)
{
void (*close_fun)(struct sock *sk, long timeout);
@@ -230,10 +275,16 @@ static void bpf_tcp_close(struct sock *sk, long timeout)
}
list_for_each_entry_safe(e, tmp, &psock->maps, list) {
- osk = cmpxchg(e->entry, sk, NULL);
- if (osk == sk) {
- list_del(&e->list);
- smap_release_sock(psock, sk);
+ if (e->entry) {
+ osk = cmpxchg(e->entry, sk, NULL);
+ if (osk == sk) {
+ list_del(&e->list);
+ smap_release_sock(psock, sk);
+ }
+ } else {
+ hlist_nulls_del_rcu(&e->hash_link->hash_node);
+ smap_release_sock(psock, e->hash_link->sk);
+ free_htab_elem(e->htab, e->hash_link);
}
}
write_unlock_bh(&sk->sk_callback_lock);
@@ -1483,12 +1534,14 @@ static struct bpf_map *sock_map_alloc(union bpf_attr *attr)
return ERR_PTR(err);
}
-static void smap_list_remove(struct smap_psock *psock, struct sock **entry)
+static void smap_list_remove(struct smap_psock *psock,
+ struct sock **entry,
+ struct htab_elem *hash_link)
{
struct smap_psock_map_entry *e, *tmp;
list_for_each_entry_safe(e, tmp, &psock->maps, list) {
- if (e->entry == entry) {
+ if (e->entry == entry || e->hash_link == hash_link) {
list_del(&e->list);
break;
}
@@ -1526,7 +1579,7 @@ static void sock_map_free(struct bpf_map *map)
* to be null and queued for garbage collection.
*/
if (likely(psock)) {
- smap_list_remove(psock, &stab->sock_map[i]);
+ smap_list_remove(psock, &stab->sock_map[i], NULL);
smap_release_sock(psock, sock);
}
write_unlock_bh(&sock->sk_callback_lock);
@@ -1585,7 +1638,7 @@ static int sock_map_delete_elem(struct bpf_map *map, void *key)
if (psock->bpf_parse)
smap_stop_sock(psock, sock);
- smap_list_remove(psock, &stab->sock_map[k]);
+ smap_list_remove(psock, &stab->sock_map[k], NULL);
smap_release_sock(psock, sock);
out:
write_unlock_bh(&sock->sk_callback_lock);
@@ -1625,7 +1678,7 @@ static int __sock_map_ctx_update_elem(struct bpf_map *map,
struct bpf_sock_progs *progs,
struct sock *sock,
struct sock **map_link,
- struct sock **hash_link,
+ struct htab_elem *hash_link,
void *key)
{
struct bpf_prog *verdict, *parse, *tx_msg;
@@ -1737,6 +1790,11 @@ static int __sock_map_ctx_update_elem(struct bpf_map *map,
e->entry = map_link;
list_add_tail(&e->list, &psock->maps);
}
+ if (hash_link) {
+ e->hash_link = hash_link;
+ e->htab = container_of(map, struct bpf_htab, map);
+ list_add_tail(&e->list, &psock->maps);
+ }
write_unlock_bh(&sock->sk_callback_lock);
return err;
out_free:
@@ -1786,7 +1844,7 @@ static int sock_map_ctx_update_elem(struct bpf_sock_ops_kern *skops,
struct smap_psock *opsock = smap_psock_sk(osock);
write_lock_bh(&osock->sk_callback_lock);
- smap_list_remove(opsock, &stab->sock_map[i]);
+ smap_list_remove(opsock, &stab->sock_map[i], NULL);
smap_release_sock(opsock, osock);
write_unlock_bh(&osock->sk_callback_lock);
}
@@ -1803,6 +1861,10 @@ int sock_map_prog(struct bpf_map *map, struct bpf_prog *prog, u32 type)
struct bpf_stab *stab = container_of(map, struct bpf_stab, map);
progs = &stab->progs;
+ } else if (map->map_type == BPF_MAP_TYPE_SOCKHASH) {
+ struct bpf_htab *htab = container_of(map, struct bpf_htab, map);
+
+ progs = &htab->progs;
} else {
return -EINVAL;
}
@@ -1863,11 +1925,19 @@ static int sock_map_update_elem(struct bpf_map *map,
static void sock_map_release(struct bpf_map *map, struct file *map_file)
{
- struct bpf_stab *stab = container_of(map, struct bpf_stab, map);
struct bpf_sock_progs *progs;
struct bpf_prog *orig;
- progs = &stab->progs;
+ if (map->map_type == BPF_MAP_TYPE_SOCKMAP) {
+ struct bpf_stab *stab = container_of(map, struct bpf_stab, map);
+
+ progs = &stab->progs;
+ } else {
+ struct bpf_htab *htab = container_of(map, struct bpf_htab, map);
+
+ progs = &htab->progs;
+ }
+
orig = xchg(&progs->bpf_parse, NULL);
if (orig)
bpf_prog_put(orig);
@@ -1880,6 +1950,396 @@ static void sock_map_release(struct bpf_map *map, struct file *map_file)
bpf_prog_put(orig);
}
+static struct bpf_map *sock_hash_alloc(union bpf_attr *attr)
+{
+ struct bpf_htab *htab;
+ int i, err;
+ u64 cost;
+
+ if (!capable(CAP_NET_ADMIN))
+ return ERR_PTR(-EPERM);
+
+ /* check sanity of attributes */
+ if (attr->max_entries == 0 ||
+ attr->map_flags & ~SOCK_CREATE_FLAG_MASK)
+ return ERR_PTR(-EINVAL);
+
+ if (attr->value_size > KMALLOC_MAX_SIZE)
+ return ERR_PTR(-E2BIG);
+
+ err = bpf_tcp_ulp_register();
+ if (err && err != -EEXIST)
+ return ERR_PTR(err);
+
+ htab = kzalloc(sizeof(*htab), GFP_USER);
+ if (!htab)
+ return ERR_PTR(-ENOMEM);
+
+ bpf_map_init_from_attr(&htab->map, attr);
+
+ htab->n_buckets = roundup_pow_of_two(htab->map.max_entries);
+ htab->elem_size = sizeof(struct htab_elem) +
+ round_up(htab->map.key_size, 8);
+
+ if (htab->n_buckets == 0 ||
+ htab->n_buckets > U32_MAX / sizeof(struct bucket))
+ goto free_htab;
+
+ cost = (u64) htab->n_buckets * sizeof(struct bucket) +
+ (u64) htab->elem_size * htab->map.max_entries;
+
+ if (cost >= U32_MAX - PAGE_SIZE)
+ goto free_htab;
+
+ htab->map.pages = round_up(cost, PAGE_SIZE) >> PAGE_SHIFT;
+ err = bpf_map_precharge_memlock(htab->map.pages);
+ if (err)
+ goto free_htab;
+
+ err = -ENOMEM;
+ htab->buckets = bpf_map_area_alloc(
+ htab->n_buckets * sizeof(struct bucket),
+ htab->map.numa_node);
+ if (!htab->buckets)
+ goto free_htab;
+
+ for (i = 0; i < htab->n_buckets; i++) {
+ INIT_HLIST_NULLS_HEAD(&htab->buckets[i].head, i);
+ raw_spin_lock_init(&htab->buckets[i].lock);
+ }
+
+ return &htab->map;
+free_htab:
+ kfree(htab);
+ return ERR_PTR(err);
+}
+
+static inline struct bucket *__select_bucket(struct bpf_htab *htab, u32 hash)
+{
+ return &htab->buckets[hash & (htab->n_buckets - 1)];
+}
+
+static inline struct hlist_nulls_head *select_bucket(struct bpf_htab *htab,
+ u32 hash)
+{
+ return &__select_bucket(htab, hash)->head;
+}
+
+static void sock_hash_free(struct bpf_map *map)
+{
+ struct bpf_htab *htab = container_of(map, struct bpf_htab, map);
+ int i;
+
+ synchronize_rcu();
+
+ /* At this point no update, lookup or delete operations can happen.
+ * However, be aware we can still get a socket state event updates,
+ * and data ready callabacks that reference the psock from sk_user_data
+ * Also psock worker threads are still in-flight. So smap_release_sock
+ * will only free the psock after cancel_sync on the worker threads
+ * and a grace period expire to ensure psock is really safe to remove.
+ */
+ rcu_read_lock();
+ for (i = 0; i < htab->n_buckets; i++) {
+ struct hlist_nulls_head *head = select_bucket(htab, i);
+ struct hlist_nulls_node *n;
+ struct htab_elem *l;
+
+ hlist_nulls_for_each_entry_safe(l, n, head, hash_node) {
+ struct sock *sock = l->sk;
+ struct smap_psock *psock;
+
+ hlist_nulls_del_rcu(&l->hash_node);
+ write_lock_bh(&sock->sk_callback_lock);
+ psock = smap_psock_sk(sock);
+ /* This check handles a racing sock event that can get
+ * the sk_callback_lock before this case but after xchg
+ * causing the refcnt to hit zero and sock user data
+ * (psock) to be null and queued for garbage collection.
+ */
+ if (likely(psock)) {
+ smap_list_remove(psock, NULL, l);
+ smap_release_sock(psock, sock);
+ }
+ write_unlock_bh(&sock->sk_callback_lock);
+ kfree(l);
+ }
+ }
+ rcu_read_unlock();
+ bpf_map_area_free(htab->buckets);
+ kfree(htab);
+}
+
+static struct htab_elem *alloc_sock_hash_elem(struct bpf_htab *htab,
+ void *key, u32 key_size, u32 hash,
+ struct sock *sk,
+ struct htab_elem *old_elem)
+{
+ struct htab_elem *l_new;
+
+ if (atomic_inc_return(&htab->count) > htab->map.max_entries) {
+ if (!old_elem) {
+ atomic_dec(&htab->count);
+ return ERR_PTR(-E2BIG);
+ }
+ }
+ l_new = kmalloc_node(htab->elem_size, GFP_ATOMIC | __GFP_NOWARN,
+ htab->map.numa_node);
+ if (!l_new)
+ return ERR_PTR(-ENOMEM);
+
+ memcpy(l_new->key, key, key_size);
+ l_new->sk = sk;
+ l_new->hash = hash;
+ return l_new;
+}
+
+static struct htab_elem *lookup_elem_raw(struct hlist_nulls_head *head,
+ u32 hash, void *key, u32 key_size)
+{
+ struct hlist_nulls_node *n;
+ struct htab_elem *l;
+
+ hlist_nulls_for_each_entry_rcu(l, n, head, hash_node)
+ if (l->hash == hash && !memcmp(&l->key, key, key_size))
+ return l;
+
+ return NULL;
+}
+
+static inline u32 htab_map_hash(const void *key, u32 key_len)
+{
+ return jhash(key, key_len, 0);
+}
+
+static struct htab_elem *lookup_nulls_elem_raw(struct hlist_nulls_head *head,
+ u32 hash, void *key,
+ u32 key_size, u32 n_buckets)
+{
+ struct hlist_nulls_node *n;
+ struct htab_elem *l;
+
+again:
+ hlist_nulls_for_each_entry_rcu(l, n, head, hash_node)
+ if (l->hash == hash && !memcmp(&l->key, key, key_size))
+ return l;
+
+ if (unlikely(get_nulls_value(n) != (hash & (n_buckets - 1))))
+ goto again;
+
+ return NULL;
+}
+
+static int sock_hash_get_next_key(struct bpf_map *map,
+ void *key, void *next_key)
+{
+ struct bpf_htab *htab = container_of(map, struct bpf_htab, map);
+ struct htab_elem *l, *next_l;
+ struct hlist_nulls_head *h;
+ u32 hash, key_size;
+ int i = 0;
+
+ WARN_ON_ONCE(!rcu_read_lock_held());
+
+ key_size = map->key_size;
+ if (!key)
+ goto find_first_elem;
+ hash = htab_map_hash(key, key_size);
+ h = select_bucket(htab, hash);
+
+ l = lookup_nulls_elem_raw(h, hash, key, key_size, htab->n_buckets);
+ if (!l)
+ goto find_first_elem;
+ next_l = hlist_nulls_entry_safe(
+ rcu_dereference_raw(hlist_nulls_next_rcu(&l->hash_node)),
+ struct htab_elem, hash_node);
+ if (next_l) {
+ memcpy(next_key, next_l->key, key_size);
+ return 0;
+ }
+
+ /* no more elements in this hash list, go to the next bucket */
+ i = hash & (htab->n_buckets - 1);
+ i++;
+
+find_first_elem:
+ /* iterate over buckets */
+ for (; i < htab->n_buckets; i++) {
+ h = select_bucket(htab, i);
+
+ /* pick first element in the bucket */
+ next_l = hlist_nulls_entry_safe(
+ rcu_dereference_raw(hlist_nulls_first_rcu(h)),
+ struct htab_elem, hash_node);
+ if (next_l) {
+ /* if it's not empty, just return it */
+ memcpy(next_key, next_l->key, key_size);
+ return 0;
+ }
+ }
+
+ /* iterated over all buckets and all elements */
+ return -ENOENT;
+}
+
+static int sock_hash_ctx_update_elem(struct bpf_sock_ops_kern *skops,
+ struct bpf_map *map,
+ void *key, u64 map_flags)
+{
+ struct bpf_htab *htab = container_of(map, struct bpf_htab, map);
+ struct bpf_sock_progs *progs = &htab->progs;
+ struct htab_elem *l_new = NULL, *l_old;
+ struct hlist_nulls_head *head;
+ struct smap_psock *psock;
+ u32 key_size, hash;
+ struct sock *sock;
+ struct bucket *b;
+ int err;
+
+ sock = skops->sk;
+
+ if (sock->sk_type != SOCK_STREAM ||
+ sock->sk_protocol != IPPROTO_TCP)
+ return -EOPNOTSUPP;
+
+ if (unlikely(map_flags > BPF_EXIST))
+ return -EINVAL;
+
+ WARN_ON_ONCE(!rcu_read_lock_held());
+ key_size = map->key_size;
+ hash = htab_map_hash(key, key_size);
+ b = __select_bucket(htab, hash);
+ head = &b->head;
+
+ err = __sock_map_ctx_update_elem(map, progs, sock, NULL, NULL, key);
+ if (err)
+ goto err;
+
+ /* bpf_map_update_elem() can be called in_irq() */
+ raw_spin_lock_bh(&b->lock);
+ l_old = lookup_elem_raw(head, hash, key, key_size);
+ if (l_old && map_flags == BPF_NOEXIST) {
+ err = -EEXIST;
+ goto bucket_err;
+ }
+ if (!l_old && map_flags == BPF_EXIST) {
+ err = -ENOENT;
+ goto bucket_err;
+ }
+
+ l_new = alloc_sock_hash_elem(htab, key, key_size, hash, sock, l_old);
+ if (IS_ERR(l_new)) {
+ err = PTR_ERR(l_new);
+ goto bucket_err;
+ }
+
+ /* add new element to the head of the list, so that
+ * concurrent search will find it before old elem
+ */
+ hlist_nulls_add_head_rcu(&l_new->hash_node, head);
+ if (l_old) {
+ psock = smap_psock_sk(l_old->sk);
+
+ hlist_nulls_del_rcu(&l_old->hash_node);
+ smap_list_remove(psock, NULL, l_old);
+ smap_release_sock(psock, l_old->sk);
+ free_htab_elem(htab, l_old);
+ }
+ raw_spin_unlock_bh(&b->lock);
+ return 0;
+bucket_err:
+ raw_spin_unlock_bh(&b->lock);
+err:
+ psock = smap_psock_sk(sock);
+ if (psock)
+ smap_release_sock(psock, sock);
+ return err;
+}
+
+static int sock_hash_update_elem(struct bpf_map *map,
+ void *key, void *value, u64 flags)
+{
+ struct bpf_sock_ops_kern skops;
+ u32 fd = *(u32 *)value;
+ struct socket *socket;
+ int err;
+
+ socket = sockfd_lookup(fd, &err);
+ if (!socket)
+ return err;
+
+ skops.sk = socket->sk;
+ if (!skops.sk) {
+ fput(socket->file);
+ return -EINVAL;
+ }
+
+ err = sock_hash_ctx_update_elem(&skops, map, key, flags);
+ fput(socket->file);
+ return err;
+}
+
+static int sock_hash_delete_elem(struct bpf_map *map, void *key)
+{
+ struct bpf_htab *htab = container_of(map, struct bpf_htab, map);
+ struct hlist_nulls_head *head;
+ struct bucket *b;
+ struct htab_elem *l;
+ u32 hash, key_size;
+ int ret = -ENOENT;
+
+ key_size = map->key_size;
+ hash = htab_map_hash(key, key_size);
+ b = __select_bucket(htab, hash);
+ head = &b->head;
+
+ raw_spin_lock_bh(&b->lock);
+ l = lookup_elem_raw(head, hash, key, key_size);
+ if (l) {
+ struct sock *sock = l->sk;
+ struct smap_psock *psock;
+
+ hlist_nulls_del_rcu(&l->hash_node);
+ write_lock_bh(&sock->sk_callback_lock);
+ psock = smap_psock_sk(sock);
+ /* This check handles a racing sock event that can get the
+ * sk_callback_lock before this case but after xchg happens
+ * causing the refcnt to hit zero and sock user data (psock)
+ * to be null and queued for garbage collection.
+ */
+ if (likely(psock)) {
+ smap_list_remove(psock, NULL, l);
+ smap_release_sock(psock, sock);
+ }
+ write_unlock_bh(&sock->sk_callback_lock);
+ free_htab_elem(htab, l);
+ ret = 0;
+ }
+ raw_spin_unlock_bh(&b->lock);
+ return ret;
+}
+
+struct sock *__sock_hash_lookup_elem(struct bpf_map *map, void *key)
+{
+ struct bpf_htab *htab = container_of(map, struct bpf_htab, map);
+ struct hlist_nulls_head *head;
+ struct htab_elem *l;
+ u32 key_size, hash;
+ struct bucket *b;
+ struct sock *sk;
+
+ key_size = map->key_size;
+ hash = htab_map_hash(key, key_size);
+ b = __select_bucket(htab, hash);
+ head = &b->head;
+
+ raw_spin_lock_bh(&b->lock);
+ l = lookup_elem_raw(head, hash, key, key_size);
+ sk = l ? l->sk : NULL;
+ raw_spin_unlock_bh(&b->lock);
+ return sk;
+}
+
const struct bpf_map_ops sock_map_ops = {
.map_alloc = sock_map_alloc,
.map_free = sock_map_free,
@@ -1890,6 +2350,16 @@ static void sock_map_release(struct bpf_map *map, struct file *map_file)
.map_release = sock_map_release,
};
+const struct bpf_map_ops sock_hash_ops = {
+ .map_alloc = sock_hash_alloc,
+ .map_free = sock_hash_free,
+ .map_lookup_elem = sock_map_lookup,
+ .map_get_next_key = sock_hash_get_next_key,
+ .map_update_elem = sock_hash_update_elem,
+ .map_delete_elem = sock_hash_delete_elem,
+ .map_release = sock_map_release,
+};
+
BPF_CALL_4(bpf_sock_map_update, struct bpf_sock_ops_kern *, bpf_sock,
struct bpf_map *, map, void *, key, u64, flags)
{
@@ -1907,3 +2377,21 @@ static void sock_map_release(struct bpf_map *map, struct file *map_file)
.arg3_type = ARG_PTR_TO_MAP_KEY,
.arg4_type = ARG_ANYTHING,
};
+
+BPF_CALL_4(bpf_sock_hash_update, struct bpf_sock_ops_kern *, bpf_sock,
+ struct bpf_map *, map, void *, key, u64, flags)
+{
+ WARN_ON_ONCE(!rcu_read_lock_held());
+ return sock_hash_ctx_update_elem(bpf_sock, map, key, flags);
+}
+
+const struct bpf_func_proto bpf_sock_hash_update_proto = {
+ .func = bpf_sock_hash_update,
+ .gpl_only = false,
+ .pkt_access = true,
+ .ret_type = RET_INTEGER,
+ .arg1_type = ARG_PTR_TO_CTX,
+ .arg2_type = ARG_CONST_MAP_PTR,
+ .arg3_type = ARG_PTR_TO_MAP_KEY,
+ .arg4_type = ARG_ANYTHING,
+};
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 5dd1dcb..878cca6 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -2088,6 +2088,13 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env,
func_id != BPF_FUNC_msg_redirect_map)
goto error;
break;
+ case BPF_MAP_TYPE_SOCKHASH:
+ if (func_id != BPF_FUNC_sk_redirect_hash &&
+ func_id != BPF_FUNC_sock_hash_update &&
+ func_id != BPF_FUNC_map_delete_elem &&
+ func_id != BPF_FUNC_msg_redirect_hash)
+ goto error;
+ break;
default:
break;
}
@@ -2124,11 +2131,14 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env,
break;
case BPF_FUNC_sk_redirect_map:
case BPF_FUNC_msg_redirect_map:
+ case BPF_FUNC_sock_map_update:
if (map->map_type != BPF_MAP_TYPE_SOCKMAP)
goto error;
break;
- case BPF_FUNC_sock_map_update:
- if (map->map_type != BPF_MAP_TYPE_SOCKMAP)
+ case BPF_FUNC_sk_redirect_hash:
+ case BPF_FUNC_msg_redirect_hash:
+ case BPF_FUNC_sock_hash_update:
+ if (map->map_type != BPF_MAP_TYPE_SOCKHASH)
goto error;
break;
default:
diff --git a/net/core/filter.c b/net/core/filter.c
index 6456e9d..d629479 100644
--- a/net/core/filter.c
+++ b/net/core/filter.c
@@ -1850,6 +1850,31 @@ int skb_do_redirect(struct sk_buff *skb)
.arg2_type = ARG_ANYTHING,
};
+BPF_CALL_4(bpf_sk_redirect_hash, struct sk_buff *, skb,
+ struct bpf_map *, map, void *, key, u64, flags)
+{
+ struct tcp_skb_cb *tcb = TCP_SKB_CB(skb);
+
+ /* If user passes invalid input drop the packet. */
+ if (unlikely(flags & ~(BPF_F_INGRESS)))
+ return SK_DROP;
+
+ tcb->bpf.flags = flags;
+ tcb->bpf.sk = __sock_hash_lookup_elem(map, key);
+
+ return SK_PASS;
+}
+
+static const struct bpf_func_proto bpf_sk_redirect_hash_proto = {
+ .func = bpf_sk_redirect_hash,
+ .gpl_only = false,
+ .ret_type = RET_INTEGER,
+ .arg1_type = ARG_PTR_TO_CTX,
+ .arg2_type = ARG_CONST_MAP_PTR,
+ .arg3_type = ARG_PTR_TO_MAP_KEY,
+ .arg4_type = ARG_ANYTHING,
+};
+
BPF_CALL_4(bpf_sk_redirect_map, struct sk_buff *, skb,
struct bpf_map *, map, u32, key, u64, flags)
{
@@ -1882,6 +1907,29 @@ struct sock *do_sk_redirect_map(struct sk_buff *skb)
.arg4_type = ARG_ANYTHING,
};
+BPF_CALL_4(bpf_msg_redirect_hash, struct sk_msg_buff *, msg,
+ struct bpf_map *, map, void *, key, u64, flags)
+{
+ /* If user passes invalid input drop the packet. */
+ if (unlikely(flags & ~(BPF_F_INGRESS)))
+ return SK_DROP;
+
+ msg->flags = flags;
+ msg->sk = __sock_hash_lookup_elem(map, key);
+
+ return SK_PASS;
+}
+
+static const struct bpf_func_proto bpf_msg_redirect_hash_proto = {
+ .func = bpf_msg_redirect_hash,
+ .gpl_only = false,
+ .ret_type = RET_INTEGER,
+ .arg1_type = ARG_PTR_TO_CTX,
+ .arg2_type = ARG_CONST_MAP_PTR,
+ .arg3_type = ARG_PTR_TO_MAP_KEY,
+ .arg4_type = ARG_ANYTHING,
+};
+
BPF_CALL_4(bpf_msg_redirect_map, struct sk_msg_buff *, msg,
struct bpf_map *, map, u32, key, u64, flags)
{
@@ -3892,6 +3940,8 @@ static unsigned long bpf_xdp_copy(void *dst_buff, const void *src_buff,
return &bpf_sock_ops_cb_flags_set_proto;
case BPF_FUNC_sock_map_update:
return &bpf_sock_map_update_proto;
+ case BPF_FUNC_sock_hash_update:
+ return &bpf_sock_hash_update_proto;
default:
return bpf_base_func_proto(func_id);
}
@@ -3903,6 +3953,8 @@ static unsigned long bpf_xdp_copy(void *dst_buff, const void *src_buff,
switch (func_id) {
case BPF_FUNC_msg_redirect_map:
return &bpf_msg_redirect_map_proto;
+ case BPF_FUNC_msg_redirect_hash:
+ return &bpf_msg_redirect_hash_proto;
case BPF_FUNC_msg_apply_bytes:
return &bpf_msg_apply_bytes_proto;
case BPF_FUNC_msg_cork_bytes:
@@ -3934,6 +3986,8 @@ static unsigned long bpf_xdp_copy(void *dst_buff, const void *src_buff,
return &bpf_get_socket_uid_proto;
case BPF_FUNC_sk_redirect_map:
return &bpf_sk_redirect_map_proto;
+ case BPF_FUNC_sk_redirect_hash:
+ return &bpf_sk_redirect_hash_proto;
default:
return bpf_base_func_proto(func_id);
}
diff --git a/tools/bpf/bpftool/map.c b/tools/bpf/bpftool/map.c
index f95fa67..2fa4cbb 100644
--- a/tools/bpf/bpftool/map.c
+++ b/tools/bpf/bpftool/map.c
@@ -67,6 +67,7 @@
[BPF_MAP_TYPE_DEVMAP] = "devmap",
[BPF_MAP_TYPE_SOCKMAP] = "sockmap",
[BPF_MAP_TYPE_CPUMAP] = "cpumap",
+ [BPF_MAP_TYPE_SOCKHASH] = "sockhash",
};
static unsigned int get_possible_cpus(void)
diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h
index 9d07465..1a19450 100644
--- a/tools/include/uapi/linux/bpf.h
+++ b/tools/include/uapi/linux/bpf.h
@@ -115,6 +115,7 @@ enum bpf_map_type {
BPF_MAP_TYPE_DEVMAP,
BPF_MAP_TYPE_SOCKMAP,
BPF_MAP_TYPE_CPUMAP,
+ BPF_MAP_TYPE_SOCKHASH,
};
enum bpf_prog_type {
@@ -821,7 +822,10 @@ struct bpf_stack_build_id {
FN(msg_apply_bytes), \
FN(msg_cork_bytes), \
FN(msg_pull_data), \
- FN(bind),
+ FN(bind), \
+ FN(sock_hash_update), \
+ FN(msg_redirect_hash), \
+ FN(sk_redirect_hash),
/* integer value in 'imm' field of BPF_CALL instruction selects which helper
* function eBPF program intends to call
diff --git a/tools/testing/selftests/bpf/bpf_helpers.h b/tools/testing/selftests/bpf/bpf_helpers.h
index d8223d9..fc6cf08 100644
--- a/tools/testing/selftests/bpf/bpf_helpers.h
+++ b/tools/testing/selftests/bpf/bpf_helpers.h
@@ -73,7 +73,7 @@ static int (*bpf_getsockopt)(void *ctx, int level, int optname, void *optval,
(void *) BPF_FUNC_getsockopt;
static int (*bpf_sock_ops_cb_flags_set)(void *ctx, int flags) =
(void *) BPF_FUNC_sock_ops_cb_flags_set;
-static int (*bpf_sk_redirect_map)(void *ctx, void *map, int key, int flags) =
+static int (*bpf_sk_redirect_map)(void *ctx, void *map, void *key, int flags) =
(void *) BPF_FUNC_sk_redirect_map;
static int (*bpf_sock_map_update)(void *map, void *key, void *value,
unsigned long long flags) =
@@ -86,7 +86,7 @@ static int (*bpf_perf_prog_read_value)(void *ctx, void *buf,
(void *) BPF_FUNC_perf_prog_read_value;
static int (*bpf_override_return)(void *ctx, unsigned long rc) =
(void *) BPF_FUNC_override_return;
-static int (*bpf_msg_redirect_map)(void *ctx, void *map, int key, int flags) =
+static int (*bpf_msg_redirect_map)(void *ctx, void *map, void *key, int flags) =
(void *) BPF_FUNC_msg_redirect_map;
static int (*bpf_msg_apply_bytes)(void *ctx, int len) =
(void *) BPF_FUNC_msg_apply_bytes;
@@ -96,6 +96,15 @@ static int (*bpf_msg_pull_data)(void *ctx, int start, int end, int flags) =
(void *) BPF_FUNC_msg_pull_data;
static int (*bpf_bind)(void *ctx, void *addr, int addr_len) =
(void *) BPF_FUNC_bind;
+static int (*bpf_sock_hash_update)(void *map, void *key, void *value,
+ unsigned long long flags) =
+ (void *) BPF_FUNC_sock_hash_update;
+static int (*bpf_msg_redirect_hash)(void *map, void *key, void *value,
+ unsigned long long flags) =
+ (void *) BPF_FUNC_msg_redirect_hash;
+static int (*bpf_sk_redirect_hash)(void *map, void *key, void *value,
+ unsigned long long flags) =
+ (void *) BPF_FUNC_sk_redirect_hash;
/* llvm builtin functions that eBPF C program may use to
* emit BPF_LD_ABS and BPF_LD_IND instructions
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox