* Re: [Patch v4 net-next 04/12] net: fec: parser max queue number from dt file
From: Zhi Li @ 2014-09-15 13:21 UTC (permalink / raw)
To: Lothar Waßmann
Cc: Frank.Li@freescale.com, Duan Fugang-B38611, David Miller,
netdev@vger.kernel.org, devicetree@vger.kernel.org,
Russell King - ARM Linux, Shawn Guo,
linux-arm-kernel@lists.infradead.org
In-Reply-To: <20140915093926.11bdbc32@ipc1.ka-ro>
On Mon, Sep 15, 2014 at 2:39 AM, Lothar Waßmann <LW@karo-electronics.de> wrote:
> Hi,
>
> Frank.Li@freescale.com wrote:
>> From: Fugang Duan <B38611@freescale.com>
>>
>> By default, the tx/rx queue number is 1, user can config the queue number
>> at DTS file like this:
>> fsl,num-tx-queues=<3>;
>> fsl,num-rx-queues=<3>
>>
>> Since i.MX6SX enet-AVB IP support multi queues, so use multi queues
>> interface to allocate and set up an Ethernet device.
>>
>> Signed-off-by: Fugang Duan <B38611@freescale.com>
>> Signed-off-by: Frank Li <Frank.Li@freescale.com>
>> ---
>> drivers/net/ethernet/freescale/fec.h | 2 ++
>> drivers/net/ethernet/freescale/fec_main.c | 46 ++++++++++++++++++++++++++++++-
>> 2 files changed, 47 insertions(+), 1 deletion(-)
>>
>> diff --git a/drivers/net/ethernet/freescale/fec.h b/drivers/net/ethernet/freescale/fec.h
>> index b2b91f8..72fb90f 100644
>> --- a/drivers/net/ethernet/freescale/fec.h
>> +++ b/drivers/net/ethernet/freescale/fec.h
>> @@ -356,6 +356,8 @@ struct fec_enet_private {
>>
>> bool ptp_clk_on;
>> struct mutex ptp_clk_mutex;
>> + unsigned int num_tx_queues;
>> + unsigned int num_rx_queues;
>>
>> /* The saved address of a sent-in-place packet/buffer, for skfree(). */
>> struct fec_enet_priv_tx_q *tx_queue[FEC_ENET_MAX_TX_QS];
>> diff --git a/drivers/net/ethernet/freescale/fec_main.c b/drivers/net/ethernet/freescale/fec_main.c
>> index 4c0d2ee..2240df0 100644
>> --- a/drivers/net/ethernet/freescale/fec_main.c
>> +++ b/drivers/net/ethernet/freescale/fec_main.c
>> @@ -2709,6 +2709,42 @@ static void fec_reset_phy(struct platform_device *pdev)
>> }
>> #endif /* CONFIG_OF */
>>
>> +static void
>> +fec_enet_get_queue_num(struct platform_device *pdev, int *num_tx, int *num_rx)
>> +{
>> + struct device_node *np = pdev->dev.of_node;
>> + int err;
>> +
>> + *num_tx = *num_rx = 1;
>> +
>> + if (!np || !of_device_is_available(np))
>> + return;
>> +
>> + /* parse the num of tx and rx queues */
>> + err = of_property_read_u32(np, "fsl,num-tx-queues", num_tx);
>> + err |= of_property_read_u32(np, "fsl,num-rx-queues", num_rx);
>> + if (err) {
>> + *num_tx = 1;
>> + *num_rx = 1;
> Shouldn't these be handled seperately? So that if only one of those
> properties is given in DT the specified value for that property should
> be used?
>
>> + return;
>> + }
>> +
>> + if (*num_tx < 1 || *num_tx > FEC_ENET_MAX_TX_QS) {
>> + dev_err(&pdev->dev, "Invalidate num_tx(=%d), fail back to 1\n",
>>
> s/Invalidate/Invalid/
> s/fail/fall/
> If the problem is fixed up by the driver rather than leading to an
> error exit it should be dev_warn() rather than dev_err().
>
>> + *num_tx);
>> + *num_tx = 1;
>> + return;
>> + }
>> +
>> + if (*num_rx < 1 || *num_rx > FEC_ENET_MAX_RX_QS) {
>> + dev_err(&pdev->dev, "Invalidate num_rx(=%d), fail back to 1\n",
> dto.
>
>> + *num_rx);
>> + *num_rx = 1;
>> + return;
>> + }
>> +
>> +}
>> +
>> static int
>> fec_probe(struct platform_device *pdev)
>> {
>> @@ -2720,13 +2756,18 @@ fec_probe(struct platform_device *pdev)
>> const struct of_device_id *of_id;
>> static int dev_id;
>> struct device_node *np = pdev->dev.of_node, *phy_node;
>> + int num_tx_qs = 1;
>> + int num_rx_qs = 1;
>>
> useless initialization. These variables are initialized in
> fec_enet_get_queue_num() anyway.
David have applied the whole patch serials.
How to handle it? use new patch?
best regards
Frank Li
>
>> of_id = of_match_device(fec_dt_ids, &pdev->dev);
>> if (of_id)
>> pdev->id_entry = of_id->data;
>>
>> + fec_enet_get_queue_num(pdev, &num_tx_qs, &num_rx_qs);
>> +
>
>
> Lothar Waßmann
> --
> ___________________________________________________________
>
> Ka-Ro electronics GmbH | Pascalstraße 22 | D - 52076 Aachen
> Phone: +49 2408 1402-0 | Fax: +49 2408 1402-10
> Geschäftsführer: Matthias Kaussen
> Handelsregistereintrag: Amtsgericht Aachen, HRB 4996
>
> www.karo-electronics.de | info@karo-electronics.de
> ___________________________________________________________
^ permalink raw reply
* Re: [PATCH v2 net-next 7/7] gre: TX path for GRE/UDP foo-over-udp encapsulation
From: Or Gerlitz @ 2014-09-15 13:18 UTC (permalink / raw)
To: Tom Herbert; +Cc: davem, netdev
On Mon, Sep 15, 2014 at 6:08 AM, Tom Herbert <therbert@google.com> wrote:
Empty change log here and also on patches 5 and 6, can you please add
it in v3 and also reply here with short write up? I assume the text
for patches 5/6/7 would be similar so for V2 enough if you just reply
here.
> Signed-off-by: Tom Herbert <therbert@google.com>
> ---
> include/uapi/linux/if_tunnel.h | 4 ++
> net/ipv4/ip_gre.c | 98 +++++++++++++++++++++++++++++++++++++++---
> 2 files changed, 95 insertions(+), 7 deletions(-)
>
> diff --git a/include/uapi/linux/if_tunnel.h b/include/uapi/linux/if_tunnel.h
> index 9fedca7..7c832af 100644
> --- a/include/uapi/linux/if_tunnel.h
> +++ b/include/uapi/linux/if_tunnel.h
> @@ -106,6 +106,10 @@ enum {
> IFLA_GRE_ENCAP_LIMIT,
> IFLA_GRE_FLOWINFO,
> IFLA_GRE_FLAGS,
> + IFLA_GRE_ENCAP_TYPE,
> + IFLA_GRE_ENCAP_FLAGS,
> + IFLA_GRE_ENCAP_SPORT,
> + IFLA_GRE_ENCAP_DPORT,
> __IFLA_GRE_MAX,
> };
>
> diff --git a/net/ipv4/ip_gre.c b/net/ipv4/ip_gre.c
> index 9b84254..5681344 100644
> --- a/net/ipv4/ip_gre.c
> +++ b/net/ipv4/ip_gre.c
> @@ -239,7 +239,7 @@ static void __gre_xmit(struct sk_buff *skb, struct net_device *dev,
> tpi.seq = htonl(tunnel->o_seqno);
>
> /* Push GRE header. */
> - gre_build_header(skb, &tpi, tunnel->hlen);
> + gre_build_header(skb, &tpi, tunnel->tun_hlen);
>
> ip_tunnel_xmit(skb, dev, tnl_params, tnl_params->protocol);
> }
If I got it right, this hunk is the only part of the patch that deals
with the TX path and the test is setup, right? would be good to
reflect that in the patch title, maybe something like "Setup and TX
path for [...]" ditto for patches 5/6
Or
^ permalink raw reply
* [PATCH net] sfc: fix addr_list_lock spinlock use before init
From: Nikolay Aleksandrov @ 2014-09-15 12:55 UTC (permalink / raw)
To: netdev
Cc: davem, Nikolay Aleksandrov, Ben Hutchings, Shradha Shah,
Solarflare linux maintainers
When the module is initializing the ports it may call a function that
uses addr_list_lock before register_netdevice() has been called for that
port i.e. addr_list_lock is still uninitialized. The function in
question is efx_farch_filter_sync_rx_mode(), now it looks pointless to call
it before the port has been registered so alter the reconfigure_mac
callbacks to check if the port has been registered using the existing
efx_dev_registered() macro.
An example calltrace is (taken from net-next, but it's the same in net):
[ 79.454689] BUG: spinlock bad magic on CPU#1, insmod/9650
[ 79.455046] lock: 0xffff88005a61a250, .magic: 00000000, .owner:
<none>/-1, .owner_cpu: 0
[ 79.455488] CPU: 1 PID: 9650 Comm: insmod Tainted: G O
3.17.0-rc4+ #30
[ 79.455937] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[ 79.456243] 0000000000000000 00000000a604ae4a ffff8800048239f8
ffffffff81731149
[ 79.457476] 0000000000000000 ffff880004823a18 ffffffff8172d242
ffff88005a61a250
[ 79.458126] ffffffff81a39357 ffff880004823a38 ffffffff8172d26d
ffff88005a61a250
[ 79.458761] Call Trace:
[ 79.458987] [<ffffffff81731149>] dump_stack+0x4d/0x66
[ 79.459259] [<ffffffff8172d242>] spin_dump+0x91/0x96
[ 79.459526] [<ffffffff8172d26d>] spin_bug+0x26/0x2b
[ 79.459788] [<ffffffff810d7e67>] do_raw_spin_lock+0x127/0x140
[ 79.460069] [<ffffffff8173812a>] _raw_spin_lock_bh+0x1a/0x20
[ 79.460348] [<ffffffffa0203872>]
efx_farch_filter_sync_rx_mode+0x32/0x100 [sfc]
[ 79.460790] [<ffffffffa0209735>] siena_mac_reconfigure+0x25/0xc0
[sfc]
[ 79.461087] [<ffffffffa020a1c2>] ? siena_init_nic+0x2b2/0x2c0 [sfc]
[ 79.461374] [<ffffffffa01fb7c5>] efx_pci_probe+0xb45/0x13c0 [sfc]
[ 79.461659] [<ffffffff813b8425>] local_pci_probe+0x45/0xa0
[ 79.461928] [<ffffffff813b9615>] ? pci_match_device+0xe5/0x110
[ 79.462211] [<ffffffff813b9779>] pci_device_probe+0xf9/0x150
[ 79.462486] [<ffffffff8148b93d>] driver_probe_device+0x12d/0x3d0
[ 79.462767] [<ffffffff8148bcb3>] __driver_attach+0x93/0xa0
[ 79.463043] [<ffffffff8148bc20>] ? __device_attach+0x40/0x40
[ 79.463316] [<ffffffff814897b3>] bus_for_each_dev+0x73/0xc0
[ 79.463590] [<ffffffff8148b33e>] driver_attach+0x1e/0x20
[ 79.463857] [<ffffffff8148af20>] bus_add_driver+0x180/0x250
[ 79.464137] [<ffffffffa023f000>] ? 0xffffffffa023f000
[ 79.464401] [<ffffffff8148c4c4>] driver_register+0x64/0xf0
[ 79.464674] [<ffffffff813b7db0>] __pci_register_driver+0x60/0x70
[ 79.464954] [<ffffffffa023f081>] efx_init_module+0x81/0x1000 [sfc]
[ 79.465244] [<ffffffff81002144>] do_one_initcall+0xd4/0x210
[ 79.465521] [<ffffffff811c7e24>] ? __vunmap+0x94/0x100
[ 79.466652] [<ffffffff811107cc>] load_module+0x1f3c/0x2570
[ 79.466978] [<ffffffff8110c460>] ? store_uevent+0x70/0x70
[ 79.467277] [<ffffffff81208d50>] ? kernel_read+0x50/0x80
[ 79.467554] [<ffffffff81110fd6>] SyS_finit_module+0xa6/0xd0
[ 79.467829] [<ffffffff81738929>] system_call_fastpath+0x16/0x1b
CC: Ben Hutchings <ben@decadent.org.uk>
CC: Shradha Shah <sshah@solarflare.com>
CC: Solarflare linux maintainers <linux-net-drivers@solarflare.com>
Fixes: 964e61355e94 ("sfc: Cleanup Falcon-arch simple MAC filter state")
Signed-off-by: Nikolay Aleksandrov <nikolay@redhat.com>
---
drivers/net/ethernet/sfc/falcon.c | 3 ++-
drivers/net/ethernet/sfc/siena.c | 3 ++-
2 files changed, 4 insertions(+), 2 deletions(-)
diff --git a/drivers/net/ethernet/sfc/falcon.c b/drivers/net/ethernet/sfc/falcon.c
index 157037546d30..f8cff9ce3bc1 100644
--- a/drivers/net/ethernet/sfc/falcon.c
+++ b/drivers/net/ethernet/sfc/falcon.c
@@ -1210,7 +1210,8 @@ static int falcon_reconfigure_xmac(struct efx_nic *efx)
{
struct falcon_nic_data *nic_data = efx->nic_data;
- efx_farch_filter_sync_rx_mode(efx);
+ if (efx_dev_registered(efx))
+ efx_farch_filter_sync_rx_mode(efx);
falcon_reconfigure_xgxs_core(efx);
falcon_reconfigure_xmac_core(efx);
diff --git a/drivers/net/ethernet/sfc/siena.c b/drivers/net/ethernet/sfc/siena.c
index ae696855f21a..7d77590d09c9 100644
--- a/drivers/net/ethernet/sfc/siena.c
+++ b/drivers/net/ethernet/sfc/siena.c
@@ -593,7 +593,8 @@ static int siena_mac_reconfigure(struct efx_nic *efx)
MC_CMD_SET_MCAST_HASH_IN_HASH0_OFST +
sizeof(efx->multicast_hash));
- efx_farch_filter_sync_rx_mode(efx);
+ if (efx_dev_registered(efx))
+ efx_farch_filter_sync_rx_mode(efx);
WARN_ON(!mutex_is_locked(&efx->mac_lock));
--
1.9.3
^ permalink raw reply related
* Re: [net-next v7 0/4] Refactor vxlan and l2tp to use common UDP tunnel APIs
From: Or Gerlitz @ 2014-09-15 12:54 UTC (permalink / raw)
To: Andy Zhou; +Cc: davem, netdev
In-Reply-To: <1410777267-28237-1-git-send-email-azhou@nicira.com>
On Mon, Sep 15, 2014 at 1:34 PM, Andy Zhou <azhou@nicira.com> wrote:
Hi Andy,
It's a bit hard to track/review a series with empty cover letter and
no Vn-1 --> Vn change tracking... can you please
add it? no need to re-spin V8 just for that, you can reply here. BTW
You did had non-empty cover letter in V5
Or.
> Andy Zhou (4):
> udp_tunnel: Seperate ipv6 functions into its own file.
> udp-tunnel: Expand UDP tunnel APIs
> vxlan: Refactor vxlan driver to make use of the common UDP tunnel
> functions.
> l2tp: Refactor l2tp core driver to make use of the common UDP tunnel
> functions
>
> drivers/net/vxlan.c | 105 ++++++++--------------------------
> include/net/udp_tunnel.h | 83 ++++++++++++++++++++++++++-
> net/ipv4/udp_tunnel.c | 138 ++++++++++++++++++++++++---------------------
> net/ipv6/Makefile | 1 +
> net/ipv6/ip6_udp_tunnel.c | 105 ++++++++++++++++++++++++++++++++++
> net/l2tp/l2tp_core.c | 24 ++++----
> 6 files changed, 292 insertions(+), 164 deletions(-)
> create mode 100644 net/ipv6/ip6_udp_tunnel.c
>
> --
> 1.7.9.5
>
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH 1/5] rhashtable: Remove gfp_flags from insert and remove functions
From: Thomas Graf @ 2014-09-15 12:49 UTC (permalink / raw)
To: Eric Dumazet
Cc: davem, paulmck, john.r.fastabend, kaber, netdev, linux-kernel
In-Reply-To: <1410784535.7106.164.camel@edumazet-glaptop2.roam.corp.google.com>
On 09/15/14 at 05:35am, Eric Dumazet wrote:
> On Mon, 2014-09-15 at 14:18 +0200, Thomas Graf wrote:
> > As the expansion/shrinking is moved to a worker thread, no allocations
> > will be performed anymore.
> >
>
> You meant : no GFP_ATOMIC allocations ?
>
> I would rephrase using something like :
>
> Because hash resizes are potentially time consuming, they'll be
> performed in process context where GFP_KERNEL allocations are preferred.
I meant to say no allocations in insert/remove anymore but your wording
is even clearer. I'll update it.
> > - tbl = kzalloc(size, flags);
> > + tbl = kzalloc(size, GFP_KERNEL);
>
> Add __GFP_NOWARN, as you fallback to vzalloc ?
Good point.
> btw, inet hash table uses alloc_large_system_hash() which spreads the
> pages on all available NUMA nodes :
>
> # dmesg | grep "TCP established"
> [ 1.223046] TCP established hash table entries: 524288 (order: 10, 4194304 bytes)
> # grep alloc_large_system_hash /proc/vmallocinfo | grep pages=1024
> 0xffffc900181d6000-0xffffc900185d7000 4198400 alloc_large_system_hash+0x177/0x237 pages=1024 vmalloc vpages N0=512 N1=512
> 0xffffc900185d7000-0xffffc900189d8000 4198400 alloc_large_system_hash+0x177/0x237 pages=1024 vmalloc vpages N0=512 N1=512
> 0xffffc90028a01000-0xffffc90028e02000 4198400 alloc_large_system_hash+0x177/0x237 pages=1024 vmalloc vpages N0=512 N1=512
>
> So we might keep in mind to keep this numa policy.
Agreed. Will introduce this through a table parameter option when
converting the inet hash table.
^ permalink raw reply
* Re: [patch net-next 00/13] introduce rocker switch driver with openvswitch hardware accelerated datapath
From: Thomas Graf @ 2014-09-15 12:43 UTC (permalink / raw)
To: Alexei Starovoitov
Cc: Jiri Pirko, netdev, davem, nhorman, andy, dborkman, ogerlitz,
jesse, pshelar, azhou, ben, stephen, jeffrey.t.kirsher, vyasevic,
xiyou.wangcong, john.r.fastabend, edumazet, jhs, sfeldma,
f.fainelli, roopa, linville, dev, jasowang, ebiederm,
nicolas.dichtel, ryazanov.s.a, buytenh, aviadr, nbd, Neil.Jerram,
ronye
In-Reply-To: <20140909210910.GA25899@ITs-MacBook-Pro.local>
On 09/09/14 at 02:09pm, Alexei Starovoitov wrote:
> On Mon, Sep 08, 2014 at 02:54:13PM +0100, Thomas Graf wrote:
> > [0] https://docs.google.com/document/d/195waUliu7G5YYVuXHmLmHgJ38DFSte321WPq0oaFhyU/edit?usp=sharing
> > (Publicly accessible and open for comments)
>
> Great doc. Very clear. I wish I could write docs like this :)
>
> Few questions:
> - on the 1st slide dpdk is used accept vm and lxc packet. How is that working?
> I know of 3 dpdk mechanisms to receive vm traffic, but all of them are kinda
> deficient, since offloads need to be disabled inside VM, so VM to VM
> performance over dpdk is not impressive. What is there for lxc?
> Is there a special pmd that can take packets from veth?
Glad to see you are paying attention ;-) I'm assuming somebody to
write a veth PMD at some point. It does not exist yet.
> - full offload vs partial.
> The doc doesn't say, but I suspect we want transition from full to partial
> to be transparent? Especially for lxc. criu should be able to snapshot
> container on one box with full offload and restore it seamlessly on the
> other machine with partial offload, right?
Correct. In a full offload environment, the CPU path could still use
partial offload functionality. I'll update the doc.
> - full offload with two nics.
> how bonding and redundancy suppose to work in such case?
> If wire attached to eth0 no longer passing packet, how traffic from VM1
> will reach eth1 on a different nic? Via sw datapath (flow table) ?
Yes.
> I suspect we want to reuse current bonding/team abstraction here.
Yes, both kernel bond/team and OVS group table based abstraction (see
Simon's recent effrots).
> I'm not quite getting the whole point of two separate physical nics.
> Is it for completeness and generality of the picture ?
Correct. It is entirely to outline the more difficult case of multiple
physical NICs.
> I think typical hypervisor will likely have only one multi-port nic, then
> bonding can be off-loaded within single nic via bonding driver.
Agreed. I would expect that to be the reference architecture.
> Partial offload scenario doesn't have this issue, since 'flow table'
> is fed by standard netdev which can be bond-dev and everything else, right?
It is unclear how a virtual LAG device would forward flow table hints.
A particular NIC might take 5 tuples which point to a particular fwd
entry. The state of this offload might be different on individual NICs
forming a bond. Either case, it should be abstracted.
> - number of VFs
> I believe it's still very limited even in the newest nics, but
> number of containers will be large.
Agreed.
> So some lxcs will be using VFs and some will use standard veth?
Likely yes. I could forsee this to be driven by an elephant detection
algorithm or driven by policy.
> We cannot swap them dynamically based on load, so I'm not sure
> how VF approach is generically applicable here. For some use cases
> with demanding lxcs, it probably helps, but is it worth the gains?
TBH, I don't know. I'm trying to figure that out ;-) It is obvious that
dedicating individual cores to PMDs is not ideal either for lxc type
workloads. The same question also applies to live migration which is
complicated by this and DPDK type setups. However, I believe that a
proper HW offload abstraction API is superior in terms of providing
virtualization abstraction but I'm afraid we won't know for sure until
we've actually tried it ;-)
^ permalink raw reply
* RE: [PATCH v2 3/4] net: stmmac: add support for Intel Quark X1000
From: Kweh, Hock Leong @ 2014-09-15 12:42 UTC (permalink / raw)
To: David Miller
Cc: peppe.cavallaro@st.com, rayagond@vayavyalabs.com,
vbridgers2013@gmail.com, srinivas.kandagatla@st.com,
wens@csie.org, netdev@vger.kernel.org,
linux-kernel@vger.kernel.org, Ong, Boon Leong
In-Reply-To: <20140912.181423.1831537443587796533.davem@davemloft.net>
> -----Original Message-----
> From: David Miller [mailto:davem@davemloft.net]
> Sent: Saturday, September 13, 2014 6:14 AM
> From: Kweh Hock Leong <hock.leong.kweh@intel.com>
> Date: Thu, 11 Sep 2014 16:38:39 +0800
>
> > + if ((!strcmp(quark_x1000_phy_info[i].board_name,
> board_name)) &&
> > + quark_x1000_phy_info[i].pci_func_num == func_num)
>
> It is entirely erroneous to identify a device by it's _PHYSICAL_ geographic
> location on the PCI bus.
>
> Please get rid of this PCI function number comparison and if necessary find
> another means of identification.
Hi David,
Here is some background of this work. Intel Quark X1000 has 2 stmmac Ethernet IP
built in the SoC. They both are using the same PCI DEVICE ID number. The only things
to differentiate them is PCI BUS DEVICE FUNCTION (Bus:Dev:Func) number which are
fix numbers 00:20:6 for port 1 and 00:20:7 for port 2 stated in Quark X1000 datasheet.
(https://communities.intel.com/docs/DOC-23092 page 44 & 97)
When I was looking into making the code to upstream, I do think about is there a better
identification way to do it? But, my mind still brought me back to this PCI FUNC number.
So, i would like to understand the concern of using PCI FUNC number and also would like
to see is there any advices, suggestion or pointer to deal with the scenario here.
Appreciate to the comments sharing. Thanks.
Regards,
Wilson
^ permalink raw reply
* Re: [PATCH net-next 04/14] tipc: add sock dump to new netlink api
From: Richard Alpe @ 2014-09-15 12:35 UTC (permalink / raw)
To: Florian Westphal; +Cc: David Miller, netdev, tipc-discussion
In-Reply-To: <20140915085152.GC14006@breakpoint.cc>
On 09/15/2014 10:51 AM, Florian Westphal wrote:
> Richard Alpe <richard.alpe@ericsson.com> wrote:
>>> You can't just say sometimes you'll partially list the set of nested
>>> attributes in an object, you must public the entire object fully in
>>> the netlink message or skip the object entirely.
>> Ok. I bluntly assumed we could put some reassemble logic in the
>> client as the end integrity should still be preserved(?).
>>
>>> I would suggest that you instead size the amount of space you'll
>>> need for at least the first socket being listed, and if NLMSG_GOODSIZE
>>> is insufficient, allocate as much as you will actually need.
>>>
>>> Then you put full socket netlink blobs in there, including all nested
>>> attributes, and then stop and reset back the the most recent full socket
>>> published if you run out of space.
>> The amount of publications a socket can have is large (~65 000). Do
>> you still think this a viable solution?
>
> I suggest to look at nf_conntrack_netlink.c ctnetlink_dump_table() and
> ctnetlink_fill_info().
>
> It should be doing something similar to what you want and it handles
> the restarts correctly, i.e., cancels all partial nested attributes
> on error and resumes at the beginning of said entry on the next dump.
Thanks. In the case of ctnetlink_fill_info() it looks to me as if a set
of nested attributes is sure to fit inside a new message and that you
only have to cancel if the accumulation of multiple top level entities
along where their nested attributes fills the message(?).
The problem in this case isn't solely the listing of a large amount of
publication attributes. The problem is that there can be an arbitrary
amount of sockets with an arbitrary amount of associated (nested)
publications (~65 000 max).
I "solved" this by nesting as many complete publications as possible for
each socket. This "works" but as David points out the nested publication
data for a specific socket might or might not be complete. This is
solely indicated by the NLMSG_DONE flag, forcing the client to know this
and reassemble.
I'm now considering splitting the socket and publication listing so that
the client will have to ask for a list of sockets and then ask for there
associated publications individually. This would reduce the complexity
on the kernel side but it may introduce some additional overhead when
the socket count is high. On the other hand it opens the opportunity for
client to only ask for publications.
Cheers
Richard
^ permalink raw reply
* Re: [PATCH 1/5] rhashtable: Remove gfp_flags from insert and remove functions
From: Eric Dumazet @ 2014-09-15 12:35 UTC (permalink / raw)
To: Thomas Graf; +Cc: davem, paulmck, john.r.fastabend, kaber, netdev, linux-kernel
In-Reply-To: <bd3aa4282e89b022620225101ab491e344852f5f.1410782841.git.tgraf@suug.ch>
On Mon, 2014-09-15 at 14:18 +0200, Thomas Graf wrote:
> As the expansion/shrinking is moved to a worker thread, no allocations
> will be performed anymore.
>
You meant : no GFP_ATOMIC allocations ?
I would rephrase using something like :
Because hash resizes are potentially time consuming, they'll be
performed in process context where GFP_KERNEL allocations are preferred.
> Signed-off-by: Thomas Graf <tgraf@suug.ch>
> ---
> include/linux/rhashtable.h | 10 +++++-----
> lib/rhashtable.c | 41 +++++++++++++++++------------------------
> net/netfilter/nft_hash.c | 4 ++--
> net/netlink/af_netlink.c | 4 ++--
> 4 files changed, 26 insertions(+), 33 deletions(-)
>
> diff --git a/include/linux/rhashtable.h b/include/linux/rhashtable.h
> index fb298e9d..942fa44 100644
> --- a/include/linux/rhashtable.h
> +++ b/include/linux/rhashtable.h
> @@ -96,16 +96,16 @@ int rhashtable_init(struct rhashtable *ht, struct rhashtable_params *params);
> u32 rhashtable_hashfn(const struct rhashtable *ht, const void *key, u32 len);
> u32 rhashtable_obj_hashfn(const struct rhashtable *ht, void *ptr);
>
> -void rhashtable_insert(struct rhashtable *ht, struct rhash_head *node, gfp_t);
> -bool rhashtable_remove(struct rhashtable *ht, struct rhash_head *node, gfp_t);
> +void rhashtable_insert(struct rhashtable *ht, struct rhash_head *node);
> +bool rhashtable_remove(struct rhashtable *ht, struct rhash_head *node);
> void rhashtable_remove_pprev(struct rhashtable *ht, struct rhash_head *obj,
> - struct rhash_head __rcu **pprev, gfp_t flags);
> + struct rhash_head __rcu **pprev);
>
> bool rht_grow_above_75(const struct rhashtable *ht, size_t new_size);
> bool rht_shrink_below_30(const struct rhashtable *ht, size_t new_size);
>
> -int rhashtable_expand(struct rhashtable *ht, gfp_t flags);
> -int rhashtable_shrink(struct rhashtable *ht, gfp_t flags);
> +int rhashtable_expand(struct rhashtable *ht);
> +int rhashtable_shrink(struct rhashtable *ht);
>
> void *rhashtable_lookup(const struct rhashtable *ht, const void *key);
> void *rhashtable_lookup_compare(const struct rhashtable *ht, u32 hash,
> diff --git a/lib/rhashtable.c b/lib/rhashtable.c
> index 8dfec3f..c133d82 100644
> --- a/lib/rhashtable.c
> +++ b/lib/rhashtable.c
> @@ -108,13 +108,13 @@ static u32 head_hashfn(const struct rhashtable *ht,
> return obj_hashfn(ht, rht_obj(ht, he), hsize);
> }
>
> -static struct bucket_table *bucket_table_alloc(size_t nbuckets, gfp_t flags)
> +static struct bucket_table *bucket_table_alloc(size_t nbuckets)
> {
> struct bucket_table *tbl;
> size_t size;
>
> size = sizeof(*tbl) + nbuckets * sizeof(tbl->buckets[0]);
> - tbl = kzalloc(size, flags);
> + tbl = kzalloc(size, GFP_KERNEL);
Add __GFP_NOWARN, as you fallback to vzalloc ?
> if (tbl == NULL)
> tbl = vzalloc(size);
btw, inet hash table uses alloc_large_system_hash() which spreads the
pages on all available NUMA nodes :
# dmesg | grep "TCP established"
[ 1.223046] TCP established hash table entries: 524288 (order: 10, 4194304 bytes)
# grep alloc_large_system_hash /proc/vmallocinfo | grep pages=1024
0xffffc900181d6000-0xffffc900185d7000 4198400 alloc_large_system_hash+0x177/0x237 pages=1024 vmalloc vpages N0=512 N1=512
0xffffc900185d7000-0xffffc900189d8000 4198400 alloc_large_system_hash+0x177/0x237 pages=1024 vmalloc vpages N0=512 N1=512
0xffffc90028a01000-0xffffc90028e02000 4198400 alloc_large_system_hash+0x177/0x237 pages=1024 vmalloc vpages N0=512 N1=512
So we might keep in mind to keep this numa policy.
Rest of the patch looks fine to me, thanks.
^ permalink raw reply
* Re: [PATCH] iproute: update dsfield file values
From: Eric Dumazet @ 2014-09-15 12:23 UTC (permalink / raw)
To: Jesper Dangaard Brouer; +Cc: Stephen Hemminger, netdev
In-Reply-To: <20140915135248.57954fff@redhat.com>
On Mon, 2014-09-15 at 13:52 +0200, Jesper Dangaard Brouer wrote:
> On Mon, 15 Sep 2014 04:48:19 -0700
> Eric Dumazet <eric.dumazet@gmail.com> wrote:
>
> > On Mon, 2014-09-15 at 13:06 +0200, Jesper Dangaard Brouer wrote:
> >
> > > Is it a bug, that we put DSCP's Expedited Forwarding (EF) into the
> > > best-effort priority band(1) in pfifo_fast???
> > >
> >
> > With 3 bands pfifo_fast, its hard to please everyone.
> >
> > Using 5 or 6 bands would be more generic ;)
>
> We can easily fix it for EF (which should have gone into band 0).
>
> But we should likely just ignore it for AFxx.
>
Yep, please submit a patch.
^ permalink raw reply
* [PATCH 3/5] rhashtable: Convert to nulls list
From: Thomas Graf @ 2014-09-15 12:18 UTC (permalink / raw)
To: davem, eric.dumazet, paulmck, john.r.fastabend, kaber
Cc: netdev, linux-kernel
In-Reply-To: <cover.1410782841.git.tgraf@suug.ch>
In order to allow wider usage of rhashtable, use a special nulls marker
to terminate each chain. The reason for not using the existing
nulls_list is that the pprev pointer usage would not be valid as entries
can be linked in two different buckets at the same time.
Signed-off-by: Thomas Graf <tgraf@suug.ch>
---
include/linux/list_nulls.h | 3 +-
include/linux/rhashtable.h | 195 +++++++++++++++++++++++++++++++--------------
lib/rhashtable.c | 158 ++++++++++++++++++++++--------------
net/netfilter/nft_hash.c | 12 ++-
net/netlink/af_netlink.c | 9 ++-
net/netlink/diag.c | 4 +-
6 files changed, 248 insertions(+), 133 deletions(-)
diff --git a/include/linux/list_nulls.h b/include/linux/list_nulls.h
index 5d10ae36..e8c300e 100644
--- a/include/linux/list_nulls.h
+++ b/include/linux/list_nulls.h
@@ -21,8 +21,9 @@ struct hlist_nulls_head {
struct hlist_nulls_node {
struct hlist_nulls_node *next, **pprev;
};
+#define NULLS_MARKER(value) (1UL | (((long)value) << 1))
#define INIT_HLIST_NULLS_HEAD(ptr, nulls) \
- ((ptr)->first = (struct hlist_nulls_node *) (1UL | (((long)nulls) << 1)))
+ ((ptr)->first = (struct hlist_nulls_node *) NULLS_MARKER(nulls))
#define hlist_nulls_entry(ptr, type, member) container_of(ptr,type,member)
/**
diff --git a/include/linux/rhashtable.h b/include/linux/rhashtable.h
index 942fa44..e9cdbda 100644
--- a/include/linux/rhashtable.h
+++ b/include/linux/rhashtable.h
@@ -18,14 +18,12 @@
#ifndef _LINUX_RHASHTABLE_H
#define _LINUX_RHASHTABLE_H
-#include <linux/rculist.h>
+#include <linux/list_nulls.h>
struct rhash_head {
struct rhash_head __rcu *next;
};
-#define INIT_HASH_HEAD(ptr) ((ptr)->next = NULL)
-
struct bucket_table {
size_t size;
struct rhash_head __rcu *buckets[];
@@ -45,6 +43,7 @@ struct rhashtable;
* @hash_rnd: Seed to use while hashing
* @max_shift: Maximum number of shifts while expanding
* @min_shift: Minimum number of shifts while shrinking
+ * @nulls_base: Base value to generate nulls marker
* @hashfn: Function to hash key
* @obj_hashfn: Function to hash object
* @grow_decision: If defined, may return true if table should expand
@@ -59,6 +58,7 @@ struct rhashtable_params {
u32 hash_rnd;
size_t max_shift;
size_t min_shift;
+ int nulls_base;
rht_hashfn_t hashfn;
rht_obj_hashfn_t obj_hashfn;
bool (*grow_decision)(const struct rhashtable *ht,
@@ -82,6 +82,24 @@ struct rhashtable {
struct rhashtable_params p;
};
+static inline unsigned long rht_marker(const struct rhashtable *ht, u32 hash)
+{
+ return NULLS_MARKER(ht->p.nulls_base + hash);
+}
+
+#define INIT_RHT_NULLS_HEAD(ptr, ht, hash) \
+ ((ptr) = (typeof(ptr)) rht_marker(ht, hash))
+
+static inline bool rht_is_a_nulls(const struct rhash_head *ptr)
+{
+ return ((unsigned long) ptr & 1);
+}
+
+static inline unsigned long rht_get_nulls_value(const struct rhash_head *ptr)
+{
+ return ((unsigned long) ptr) >> 1;
+}
+
#ifdef CONFIG_PROVE_LOCKING
int lockdep_rht_mutex_is_held(const struct rhashtable *ht);
#else
@@ -119,92 +137,145 @@ void rhashtable_destroy(const struct rhashtable *ht);
#define rht_dereference_rcu(p, ht) \
rcu_dereference_check(p, lockdep_rht_mutex_is_held(ht))
-#define rht_entry(ptr, type, member) container_of(ptr, type, member)
-#define rht_entry_safe(ptr, type, member) \
-({ \
- typeof(ptr) __ptr = (ptr); \
- __ptr ? rht_entry(__ptr, type, member) : NULL; \
-})
+#define rht_dereference_bucket(p, tbl, hash) \
+ rcu_dereference_protected(p, lockdep_rht_mutex_is_held(ht))
+
+#define rht_dereference_bucket_rcu(p, tbl, hash) \
+ rcu_dereference_check(p, lockdep_rht_mutex_is_held(ht))
+
+#define rht_entry(tpos, pos, member) \
+ ({ tpos = container_of(pos, typeof(*tpos), member); 1; })
-#define rht_next_entry_safe(pos, ht, member) \
-({ \
- pos ? rht_entry_safe(rht_dereference((pos)->member.next, ht), \
- typeof(*(pos)), member) : NULL; \
-})
+static inline struct rhash_head *rht_get_bucket(const struct bucket_table *tbl,
+ u32 hash)
+{
+ return rht_dereference_bucket(tbl->buckets[hash], tbl, hash);
+}
+
+static inline struct rhash_head *rht_get_bucket_rcu(const struct bucket_table *tbl,
+ u32 hash)
+{
+ return rht_dereference_bucket_rcu(tbl->buckets[hash], tbl, hash);
+}
+
+/**
+ * rht_for_each_continue - continue iterating over hash chain
+ * @pos: the &struct rhash_head to use as a loop cursor.
+ * @head: the previous &struct rhash_head to continue from
+ * @tbl: the &struct bucket_table
+ * @hash: the hash value / bucket index
+ */
+#define rht_for_each_continue(pos, head, tbl, hash) \
+ for (pos = rht_dereference_bucket(head, tbl, hash); \
+ !rht_is_a_nulls(pos); \
+ pos = rht_dereference_bucket(pos->next, tbl, hash))
/**
* rht_for_each - iterate over hash chain
- * @pos: &struct rhash_head to use as a loop cursor.
- * @head: head of the hash chain (struct rhash_head *)
- * @ht: pointer to your struct rhashtable
+ * @pos: the &struct rhash_head to use as a loop cursor.
+ * @tbl: the &struct bucket_table
+ * @hash: the hash value / bucket index
*/
-#define rht_for_each(pos, head, ht) \
- for (pos = rht_dereference(head, ht); \
- pos; \
- pos = rht_dereference((pos)->next, ht))
+#define rht_for_each(pos, tbl, hash) \
+ for (pos = rht_get_bucket(tbl, hash); \
+ !rht_is_a_nulls(pos); \
+ pos = rht_dereference_bucket(pos->next, tbl, hash))
/**
* rht_for_each_entry - iterate over hash chain of given type
- * @pos: type * to use as a loop cursor.
- * @head: head of the hash chain (struct rhash_head *)
- * @ht: pointer to your struct rhashtable
- * @member: name of the rhash_head within the hashable struct.
+ * @tpos: the type * to use as a loop cursor.
+ * @pos: the &struct rhash_head to use as a loop cursor.
+ * @tbl: the &struct bucket_table
+ * @hash: the hash value / bucket index
+ * @member: name of the &struct rhash_head within the hashable struct.
*/
-#define rht_for_each_entry(pos, head, ht, member) \
- for (pos = rht_entry_safe(rht_dereference(head, ht), \
- typeof(*(pos)), member); \
- pos; \
- pos = rht_next_entry_safe(pos, ht, member))
+#define rht_for_each_entry(tpos, pos, tbl, hash, member) \
+ for (pos = rht_get_bucket(tbl, hash); \
+ (!rht_is_a_nulls(pos)) && rht_entry(tpos, pos, member); \
+ pos = rht_dereference_bucket(pos->next, tbl, hash))
/**
* rht_for_each_entry_safe - safely iterate over hash chain of given type
- * @pos: type * to use as a loop cursor.
- * @n: type * to use for temporary next object storage
- * @head: head of the hash chain (struct rhash_head *)
- * @ht: pointer to your struct rhashtable
- * @member: name of the rhash_head within the hashable struct.
+ * @tpos: the type * to use as a loop cursor.
+ * @pos: the &struct rhash_head to use as a loop cursor.
+ * @next: the &struct rhash_head to use as next in loop cursor.
+ * @tbl: the &struct bucket_table
+ * @hash: the hash value / bucket index
+ * @member: name of the &struct rhash_head within the hashable struct.
*
* This hash chain list-traversal primitive allows for the looped code to
* remove the loop cursor from the list.
*/
-#define rht_for_each_entry_safe(pos, n, head, ht, member) \
- for (pos = rht_entry_safe(rht_dereference(head, ht), \
- typeof(*(pos)), member), \
- n = rht_next_entry_safe(pos, ht, member); \
- pos; \
- pos = n, \
- n = rht_next_entry_safe(pos, ht, member))
+#define rht_for_each_entry_safe(tpos, pos, next, tbl, hash, member) \
+ for (pos = rht_get_bucket(tbl, hash), \
+ next = !rht_is_a_nulls(pos) ? \
+ rht_dereference_bucket(pos->next, tbl, hash) : \
+ (struct rhash_head *) NULLS_MARKER(0); \
+ (!rht_is_a_nulls(pos)) && rht_entry(tpos, pos, member); \
+ pos = next)
+
+/**
+ * rht_for_each_rcu_continue - continue iterating over rcu hash chain
+ * @pos: the &struct rhash_head to use as a loop cursor.
+ * @head: the previous &struct rhash_head to continue from
+ *
+ * This hash chain list-traversal primitive may safely run concurrently with
+ * the _rcu mutation primitives such as rht_insert() as long as the traversal
+ * is guarded by rcu_read_lock().
+ */
+#define rht_for_each_rcu_continue(pos, head) \
+ for (({barrier(); }), pos = rcu_dereference_raw(head); \
+ !rht_is_a_nulls(pos); \
+ pos = rcu_dereference_raw(pos->next))
/**
* rht_for_each_rcu - iterate over rcu hash chain
- * @pos: &struct rhash_head to use as a loop cursor.
- * @head: head of the hash chain (struct rhash_head *)
- * @ht: pointer to your struct rhashtable
+ * @pos: the &struct rhash_head to use as a loop cursor.
+ * @tbl: the &struct bucket_table
+ * @hash: the hash value / bucket index
+ *
+ * This hash chain list-traversal primitive may safely run concurrently with
+ * the _rcu mutation primitives such as rht_insert() as long as the traversal
+ * is guarded by rcu_read_lock().
+ */
+#define rht_for_each_rcu(pos, tbl, hash) \
+ for (({barrier(); }), pos = rht_get_bucket_rcu(tbl, hash); \
+ !rht_is_a_nulls(pos); \
+ pos = rcu_dereference_raw(pos->next))
+
+/**
+ * rht_for_each_entry_rcu_continue - continue iterating over rcu hash chain
+ * @tpos: the type * to use as a loop cursor.
+ * @pos: the &struct rhash_head to use as a loop cursor.
+ * @head: the previous &struct rhash_head to continue from
+ * @member: name of the &struct rhash_head within the hashable struct.
*
* This hash chain list-traversal primitive may safely run concurrently with
- * the _rcu fkht mutation primitives such as rht_insert() as long as the
- * traversal is guarded by rcu_read_lock().
+ * the _rcu mutation primitives such as rht_insert() as long as the traversal
+ * is guarded by rcu_read_lock().
*/
-#define rht_for_each_rcu(pos, head, ht) \
- for (pos = rht_dereference_rcu(head, ht); \
- pos; \
- pos = rht_dereference_rcu((pos)->next, ht))
+#define rht_for_each_entry_rcu_continue(tpos, pos, head, member) \
+ for (({barrier(); }), \
+ pos = rcu_dereference_raw(head); \
+ (!rht_is_a_nulls(pos)) && rht_entry(tpos, pos, member); \
+ pos = rcu_dereference_raw(pos->next))
/**
* rht_for_each_entry_rcu - iterate over rcu hash chain of given type
- * @pos: type * to use as a loop cursor.
- * @head: head of the hash chain (struct rhash_head *)
- * @member: name of the rhash_head within the hashable struct.
+ * @tpos: the type * to use as a loop cursor.
+ * @pos: the &struct rhash_head to use as a loop cursor.
+ * @tbl: the &struct bucket_table
+ * @hash: the hash value / bucket index
+ * @member: name of the &struct rhash_head within the hashable struct.
*
* This hash chain list-traversal primitive may safely run concurrently with
- * the _rcu fkht mutation primitives such as rht_insert() as long as the
- * traversal is guarded by rcu_read_lock().
+ * the _rcu mutation primitives such as rht_insert() as long as the traversal
+ * is guarded by rcu_read_lock().
*/
-#define rht_for_each_entry_rcu(pos, head, member) \
- for (pos = rht_entry_safe(rcu_dereference_raw(head), \
- typeof(*(pos)), member); \
- pos; \
- pos = rht_entry_safe(rcu_dereference_raw((pos)->member.next), \
- typeof(*(pos)), member))
+#define rht_for_each_entry_rcu(tpos, pos, tbl, hash, member) \
+ for (({barrier(); }), \
+ pos = rht_get_bucket_rcu(tbl, hash); \
+ (!rht_is_a_nulls(pos)) && rht_entry(tpos, pos, member); \
+ pos = rht_dereference_bucket_rcu(pos->next, tbl, hash))
#endif /* _LINUX_RHASHTABLE_H */
diff --git a/lib/rhashtable.c b/lib/rhashtable.c
index c10df45..d871483 100644
--- a/lib/rhashtable.c
+++ b/lib/rhashtable.c
@@ -28,6 +28,23 @@
#define HASH_DEFAULT_SIZE 64UL
#define HASH_MIN_SIZE 4UL
+/*
+ * The nulls marker consists of:
+ *
+ * +-------+-----------------------------------------------------+-+
+ * | Base | Hash |1|
+ * +-------+-----------------------------------------------------+-+
+ *
+ * Base (4 bits) : Reserved to distinguish between multiple tables.
+ * Specified via &struct rhashtable_params.nulls_base.
+ * Hash (27 bits): Full hash (unmasked) of first element added to bucket
+ * 1 (1 bit) : Nulls marker (always set)
+ *
+ */
+#define HASH_BASE_BITS 4
+#define HASH_BASE_MIN (1 << (31 - HASH_BASE_BITS))
+#define HASH_RESERVED_SPACE (HASH_BASE_BITS + 1)
+
#define ASSERT_RHT_MUTEX(HT) BUG_ON(!lockdep_rht_mutex_is_held(HT))
#ifdef CONFIG_PROVE_LOCKING
@@ -43,14 +60,22 @@ static void *rht_obj(const struct rhashtable *ht, const struct rhash_head *he)
return (void *) he - ht->p.head_offset;
}
-static u32 __hashfn(const struct rhashtable *ht, const void *key,
- u32 len, u32 hsize)
+static u32 rht_bucket_index(u32 hash, const struct bucket_table *tbl)
{
- u32 h;
+ return hash & (tbl->size - 1);
+}
- h = ht->p.hashfn(key, len, ht->p.hash_rnd);
+static u32 obj_raw_hashfn(const struct rhashtable *ht, const void *ptr)
+{
+ u32 hash;
+
+ if (unlikely(!ht->p.key_len))
+ hash = ht->p.obj_hashfn(ptr, ht->p.hash_rnd);
+ else
+ hash = ht->p.hashfn(ptr + ht->p.key_offset, ht->p.key_len,
+ ht->p.hash_rnd);
- return h & (hsize - 1);
+ return hash >> HASH_RESERVED_SPACE;
}
/**
@@ -66,23 +91,14 @@ static u32 __hashfn(const struct rhashtable *ht, const void *key,
u32 rhashtable_hashfn(const struct rhashtable *ht, const void *key, u32 len)
{
struct bucket_table *tbl = rht_dereference_rcu(ht->tbl, ht);
+ u32 hash;
- return __hashfn(ht, key, len, tbl->size);
-}
-EXPORT_SYMBOL_GPL(rhashtable_hashfn);
-
-static u32 obj_hashfn(const struct rhashtable *ht, const void *ptr, u32 hsize)
-{
- if (unlikely(!ht->p.key_len)) {
- u32 h;
-
- h = ht->p.obj_hashfn(ptr, ht->p.hash_rnd);
-
- return h & (hsize - 1);
- }
+ hash = ht->p.hashfn(key, len, ht->p.hash_rnd);
+ hash >>= HASH_RESERVED_SPACE;
- return __hashfn(ht, ptr + ht->p.key_offset, ht->p.key_len, hsize);
+ return rht_bucket_index(hash, tbl);
}
+EXPORT_SYMBOL_GPL(rhashtable_hashfn);
/**
* rhashtable_obj_hashfn - compute hash for hashed object
@@ -98,20 +114,23 @@ u32 rhashtable_obj_hashfn(const struct rhashtable *ht, void *ptr)
{
struct bucket_table *tbl = rht_dereference_rcu(ht->tbl, ht);
- return obj_hashfn(ht, ptr, tbl->size);
+ return rht_bucket_index(obj_raw_hashfn(ht, ptr), tbl);
}
EXPORT_SYMBOL_GPL(rhashtable_obj_hashfn);
static u32 head_hashfn(const struct rhashtable *ht,
- const struct rhash_head *he, u32 hsize)
+ const struct rhash_head *he,
+ const struct bucket_table *tbl)
{
- return obj_hashfn(ht, rht_obj(ht, he), hsize);
+ return rht_bucket_index(obj_raw_hashfn(ht, rht_obj(ht, he)), tbl);
}
-static struct bucket_table *bucket_table_alloc(size_t nbuckets)
+static struct bucket_table *bucket_table_alloc(struct rhashtable *ht,
+ size_t nbuckets)
{
struct bucket_table *tbl;
size_t size;
+ int i;
size = sizeof(*tbl) + nbuckets * sizeof(tbl->buckets[0]);
tbl = kzalloc(size, GFP_KERNEL);
@@ -121,6 +140,9 @@ static struct bucket_table *bucket_table_alloc(size_t nbuckets)
if (tbl == NULL)
return NULL;
+ for (i = 0; i < nbuckets; i++)
+ INIT_RHT_NULLS_HEAD(tbl->buckets[i], ht, i);
+
tbl->size = nbuckets;
return tbl;
@@ -159,34 +181,36 @@ static void hashtable_chain_unzip(const struct rhashtable *ht,
const struct bucket_table *new_tbl,
struct bucket_table *old_tbl, size_t n)
{
- struct rhash_head *he, *p, *next;
- unsigned int h;
+ struct rhash_head *he, *p;
+ struct rhash_head __rcu *next;
+ u32 hash, new_tbl_idx;
/* Old bucket empty, no work needed. */
- p = rht_dereference(old_tbl->buckets[n], ht);
- if (!p)
+ p = rht_get_bucket(old_tbl, n);
+ if (rht_is_a_nulls(p))
return;
/* Advance the old bucket pointer one or more times until it
* reaches a node that doesn't hash to the same bucket as the
* previous node p. Call the previous node p;
*/
- h = head_hashfn(ht, p, new_tbl->size);
- rht_for_each(he, p->next, ht) {
- if (head_hashfn(ht, he, new_tbl->size) != h)
+ hash = obj_raw_hashfn(ht, rht_obj(ht, p));
+ new_tbl_idx = rht_bucket_index(hash, new_tbl);
+ rht_for_each_continue(he, p->next, old_tbl, n) {
+ if (head_hashfn(ht, he, new_tbl) != new_tbl_idx)
break;
p = he;
}
- RCU_INIT_POINTER(old_tbl->buckets[n], p->next);
+ RCU_INIT_POINTER(old_tbl->buckets[n], he);
/* Find the subsequent node which does hash to the same
* bucket as node P, or NULL if no such node exists.
*/
- next = NULL;
- if (he) {
- rht_for_each(he, he->next, ht) {
- if (head_hashfn(ht, he, new_tbl->size) == h) {
- next = he;
+ INIT_RHT_NULLS_HEAD(next, ht, hash);
+ if (!rht_is_a_nulls(he)) {
+ rht_for_each_continue(he, he->next, old_tbl, n) {
+ if (head_hashfn(ht, he, new_tbl) == new_tbl_idx) {
+ next = (struct rhash_head __rcu *) he;
break;
}
}
@@ -223,7 +247,7 @@ int rhashtable_expand(struct rhashtable *ht)
if (ht->p.max_shift && ht->shift >= ht->p.max_shift)
return 0;
- new_tbl = bucket_table_alloc(old_tbl->size * 2);
+ new_tbl = bucket_table_alloc(ht, old_tbl->size * 2);
if (new_tbl == NULL)
return -ENOMEM;
@@ -239,8 +263,8 @@ int rhashtable_expand(struct rhashtable *ht)
*/
for (i = 0; i < new_tbl->size; i++) {
h = i & (old_tbl->size - 1);
- rht_for_each(he, old_tbl->buckets[h], ht) {
- if (head_hashfn(ht, he, new_tbl->size) == i) {
+ rht_for_each(he, old_tbl, h) {
+ if (head_hashfn(ht, he, new_tbl) == i) {
RCU_INIT_POINTER(new_tbl->buckets[i], he);
break;
}
@@ -268,7 +292,7 @@ int rhashtable_expand(struct rhashtable *ht)
complete = true;
for (i = 0; i < old_tbl->size; i++) {
hashtable_chain_unzip(ht, new_tbl, old_tbl, i);
- if (old_tbl->buckets[i] != NULL)
+ if (!rht_is_a_nulls(old_tbl->buckets[i]))
complete = false;
}
} while (!complete);
@@ -299,7 +323,7 @@ int rhashtable_shrink(struct rhashtable *ht)
if (ht->shift <= ht->p.min_shift)
return 0;
- ntbl = bucket_table_alloc(tbl->size / 2);
+ ntbl = bucket_table_alloc(ht, tbl->size / 2);
if (ntbl == NULL)
return -ENOMEM;
@@ -316,8 +340,9 @@ int rhashtable_shrink(struct rhashtable *ht)
* in the old table that contains entries which will hash
* to the new bucket.
*/
- for (pprev = &ntbl->buckets[i]; *pprev != NULL;
- pprev = &rht_dereference(*pprev, ht)->next)
+ for (pprev = &ntbl->buckets[i];
+ !rht_is_a_nulls(rht_dereference_bucket(*pprev, ntbl, i));
+ pprev = &rht_dereference_bucket(*pprev, ntbl, i)->next)
;
RCU_INIT_POINTER(*pprev, tbl->buckets[i + ntbl->size]);
}
@@ -350,13 +375,17 @@ EXPORT_SYMBOL_GPL(rhashtable_shrink);
void rhashtable_insert(struct rhashtable *ht, struct rhash_head *obj)
{
struct bucket_table *tbl = rht_dereference(ht->tbl, ht);
- u32 hash;
+ u32 hash, idx;
ASSERT_RHT_MUTEX(ht);
- hash = head_hashfn(ht, obj, tbl->size);
- RCU_INIT_POINTER(obj->next, tbl->buckets[hash]);
- rcu_assign_pointer(tbl->buckets[hash], obj);
+ hash = obj_raw_hashfn(ht, rht_obj(ht, obj));
+ idx = rht_bucket_index(hash, tbl);
+ if (rht_is_a_nulls(rht_get_bucket(tbl, idx)))
+ INIT_RHT_NULLS_HEAD(obj->next, ht, hash);
+ else
+ obj->next = tbl->buckets[idx];
+ rcu_assign_pointer(tbl->buckets[idx], obj);
ht->nelems++;
if (ht->p.grow_decision && ht->p.grow_decision(ht, tbl->size))
@@ -410,14 +439,13 @@ bool rhashtable_remove(struct rhashtable *ht, struct rhash_head *obj)
struct bucket_table *tbl = rht_dereference(ht->tbl, ht);
struct rhash_head __rcu **pprev;
struct rhash_head *he;
- u32 h;
+ u32 idx;
ASSERT_RHT_MUTEX(ht);
- h = head_hashfn(ht, obj, tbl->size);
-
- pprev = &tbl->buckets[h];
- rht_for_each(he, tbl->buckets[h], ht) {
+ idx = head_hashfn(ht, obj, tbl);
+ pprev = &tbl->buckets[idx];
+ rht_for_each(he, tbl, idx) {
if (he != obj) {
pprev = &he->next;
continue;
@@ -453,12 +481,12 @@ void *rhashtable_lookup(const struct rhashtable *ht, const void *key)
BUG_ON(!ht->p.key_len);
- h = __hashfn(ht, key, ht->p.key_len, tbl->size);
- rht_for_each_rcu(he, tbl->buckets[h], ht) {
+ h = rhashtable_hashfn(ht, key, ht->p.key_len);
+ rht_for_each_rcu(he, tbl, h) {
if (memcmp(rht_obj(ht, he) + ht->p.key_offset, key,
ht->p.key_len))
continue;
- return (void *) he - ht->p.head_offset;
+ return rht_obj(ht, he);
}
return NULL;
@@ -489,7 +517,7 @@ void *rhashtable_lookup_compare(const struct rhashtable *ht, u32 hash,
if (unlikely(hash >= tbl->size))
return NULL;
- rht_for_each_rcu(he, tbl->buckets[hash], ht) {
+ rht_for_each_rcu(he, tbl, hash) {
if (!compare(rht_obj(ht, he), arg))
continue;
return (void *) he - ht->p.head_offset;
@@ -560,19 +588,23 @@ int rhashtable_init(struct rhashtable *ht, struct rhashtable_params *params)
(!params->key_len && !params->obj_hashfn))
return -EINVAL;
+ if (params->nulls_base && params->nulls_base < HASH_BASE_MIN)
+ return -EINVAL;
+
params->min_shift = max_t(size_t, params->min_shift,
ilog2(HASH_MIN_SIZE));
if (params->nelem_hint)
size = rounded_hashtable_size(params);
- tbl = bucket_table_alloc(size);
+ memset(ht, 0, sizeof(*ht));
+ memcpy(&ht->p, params, sizeof(*params));
+
+ tbl = bucket_table_alloc(ht, size);
if (tbl == NULL)
return -ENOMEM;
- memset(ht, 0, sizeof(*ht));
ht->shift = ilog2(tbl->size);
- memcpy(&ht->p, params, sizeof(*params));
RCU_INIT_POINTER(ht->tbl, tbl);
if (!ht->p.hash_rnd)
@@ -652,6 +684,7 @@ static void test_bucket_stats(struct rhashtable *ht, struct bucket_table *tbl,
bool quiet)
{
unsigned int cnt, rcu_cnt, i, total = 0;
+ struct rhash_head *pos;
struct test_obj *obj;
for (i = 0; i < tbl->size; i++) {
@@ -660,7 +693,7 @@ static void test_bucket_stats(struct rhashtable *ht, struct bucket_table *tbl,
if (!quiet)
pr_info(" [%#4x/%zu]", i, tbl->size);
- rht_for_each_entry_rcu(obj, tbl->buckets[i], node) {
+ rht_for_each_entry_rcu(obj, pos, tbl, i, node) {
cnt++;
total++;
if (!quiet)
@@ -689,7 +722,8 @@ static void test_bucket_stats(struct rhashtable *ht, struct bucket_table *tbl,
static int __init test_rhashtable(struct rhashtable *ht)
{
struct bucket_table *tbl;
- struct test_obj *obj, *next;
+ struct test_obj *obj;
+ struct rhash_head *pos, *next;
int err;
unsigned int i;
@@ -755,7 +789,7 @@ static int __init test_rhashtable(struct rhashtable *ht)
error:
tbl = rht_dereference_rcu(ht->tbl, ht);
for (i = 0; i < tbl->size; i++)
- rht_for_each_entry_safe(obj, next, tbl->buckets[i], ht, node)
+ rht_for_each_entry_safe(obj, pos, next, tbl, i, node)
kfree(obj);
return err;
diff --git a/net/netfilter/nft_hash.c b/net/netfilter/nft_hash.c
index b52873c..68b654b 100644
--- a/net/netfilter/nft_hash.c
+++ b/net/netfilter/nft_hash.c
@@ -99,12 +99,13 @@ static int nft_hash_get(const struct nft_set *set, struct nft_set_elem *elem)
const struct rhashtable *priv = nft_set_priv(set);
const struct bucket_table *tbl = rht_dereference_rcu(priv->tbl, priv);
struct rhash_head __rcu * const *pprev;
+ struct rhash_head *pos;
struct nft_hash_elem *he;
u32 h;
h = rhashtable_hashfn(priv, &elem->key, set->klen);
pprev = &tbl->buckets[h];
- rht_for_each_entry_rcu(he, tbl->buckets[h], node) {
+ rht_for_each_entry_rcu(he, pos, tbl, h, node) {
if (nft_data_cmp(&he->key, &elem->key, set->klen)) {
pprev = &he->node.next;
continue;
@@ -130,7 +131,9 @@ static void nft_hash_walk(const struct nft_ctx *ctx, const struct nft_set *set,
tbl = rht_dereference_rcu(priv->tbl, priv);
for (i = 0; i < tbl->size; i++) {
- rht_for_each_entry_rcu(he, tbl->buckets[i], node) {
+ struct rhash_head *pos;
+
+ rht_for_each_entry_rcu(he, pos, tbl, i, node) {
if (iter->count < iter->skip)
goto cont;
@@ -181,12 +184,13 @@ static void nft_hash_destroy(const struct nft_set *set)
{
const struct rhashtable *priv = nft_set_priv(set);
const struct bucket_table *tbl;
- struct nft_hash_elem *he, *next;
+ struct nft_hash_elem *he;
+ struct rhash_head *pos, *next;
unsigned int i;
tbl = rht_dereference(priv->tbl, priv);
for (i = 0; i < tbl->size; i++)
- rht_for_each_entry_safe(he, next, tbl->buckets[i], priv, node)
+ rht_for_each_entry_safe(he, pos, next, tbl, i, node)
nft_hash_elem_destroy(set, he);
rhashtable_destroy(priv);
diff --git a/net/netlink/af_netlink.c b/net/netlink/af_netlink.c
index a1e6104..98e5b58 100644
--- a/net/netlink/af_netlink.c
+++ b/net/netlink/af_netlink.c
@@ -2903,7 +2903,9 @@ static struct sock *netlink_seq_socket_idx(struct seq_file *seq, loff_t pos)
const struct bucket_table *tbl = rht_dereference_rcu(ht->tbl, ht);
for (j = 0; j < tbl->size; j++) {
- rht_for_each_entry_rcu(nlk, tbl->buckets[j], node) {
+ struct rhash_head *node;
+
+ rht_for_each_entry_rcu(nlk, node, tbl, j, node) {
s = (struct sock *)nlk;
if (sock_net(s) != seq_file_net(seq))
@@ -2929,6 +2931,7 @@ static void *netlink_seq_start(struct seq_file *seq, loff_t *pos)
static void *netlink_seq_next(struct seq_file *seq, void *v, loff_t *pos)
{
+ struct rhash_head *node;
struct netlink_sock *nlk;
struct nl_seq_iter *iter;
struct net *net;
@@ -2943,7 +2946,7 @@ static void *netlink_seq_next(struct seq_file *seq, void *v, loff_t *pos)
iter = seq->private;
nlk = v;
- rht_for_each_entry_rcu(nlk, nlk->node.next, node)
+ rht_for_each_entry_rcu_continue(nlk, node, nlk->node.next, node)
if (net_eq(sock_net((struct sock *)nlk), net))
return nlk;
@@ -2955,7 +2958,7 @@ static void *netlink_seq_next(struct seq_file *seq, void *v, loff_t *pos)
const struct bucket_table *tbl = rht_dereference_rcu(ht->tbl, ht);
for (; j < tbl->size; j++) {
- rht_for_each_entry_rcu(nlk, tbl->buckets[j], node) {
+ rht_for_each_entry_rcu(nlk, node, tbl, j, node) {
if (net_eq(sock_net((struct sock *)nlk), net)) {
iter->link = i;
iter->hash_idx = j;
diff --git a/net/netlink/diag.c b/net/netlink/diag.c
index de8c74a..1062bb4 100644
--- a/net/netlink/diag.c
+++ b/net/netlink/diag.c
@@ -113,7 +113,9 @@ static int __netlink_diag_dump(struct sk_buff *skb, struct netlink_callback *cb,
req = nlmsg_data(cb->nlh);
for (i = 0; i < htbl->size; i++) {
- rht_for_each_entry(nlsk, htbl->buckets[i], ht, node) {
+ struct rhash_head *node;
+
+ rht_for_each_entry(nlsk, node, htbl, i, node) {
sk = (struct sock *)nlsk;
if (!net_eq(sock_net(sk), net))
--
1.9.3
^ permalink raw reply related
* [PATCH 2/5] rhashtable: Check for count misatch in selftest
From: Thomas Graf @ 2014-09-15 12:18 UTC (permalink / raw)
To: davem, eric.dumazet, paulmck, john.r.fastabend, kaber
Cc: netdev, linux-kernel
In-Reply-To: <cover.1410782841.git.tgraf@suug.ch>
Signed-off-by: Thomas Graf <tgraf@suug.ch>
---
lib/rhashtable.c | 19 ++++++++++++++-----
1 file changed, 14 insertions(+), 5 deletions(-)
diff --git a/lib/rhashtable.c b/lib/rhashtable.c
index c133d82..c10df45 100644
--- a/lib/rhashtable.c
+++ b/lib/rhashtable.c
@@ -648,15 +648,14 @@ static int __init test_rht_lookup(struct rhashtable *ht)
return 0;
}
-static void test_bucket_stats(struct rhashtable *ht,
- struct bucket_table *tbl,
- bool quiet)
+static void test_bucket_stats(struct rhashtable *ht, struct bucket_table *tbl,
+ bool quiet)
{
- unsigned int cnt, i, total = 0;
+ unsigned int cnt, rcu_cnt, i, total = 0;
struct test_obj *obj;
for (i = 0; i < tbl->size; i++) {
- cnt = 0;
+ rcu_cnt = cnt = 0;
if (!quiet)
pr_info(" [%#4x/%zu]", i, tbl->size);
@@ -668,6 +667,13 @@ static void test_bucket_stats(struct rhashtable *ht,
pr_cont(" [%p],", obj);
}
+ rht_for_each_rcu(pos, tbl, i)
+ rcu_cnt++;
+
+ if (rcu_cnt != cnt)
+ pr_warn("Test failed: Chain count mismach %d != %d",
+ cnt, rcu_cnt);
+
if (!quiet)
pr_cont("\n [%#x] first element: %p, chain length: %u\n",
i, tbl->buckets[i], cnt);
@@ -675,6 +681,9 @@ static void test_bucket_stats(struct rhashtable *ht,
pr_info(" Traversal complete: counted=%u, nelems=%zu, entries=%d\n",
total, ht->nelems, TEST_ENTRIES);
+
+ if (total != ht->nelems || total != TEST_ENTRIES)
+ pr_warn("Test failed: Total count mismatch ^^^");
}
static int __init test_rhashtable(struct rhashtable *ht)
--
1.9.3
^ permalink raw reply related
* [PATCH 4/5] spinlock: Add spin_lock_bh_nested()
From: Thomas Graf @ 2014-09-15 12:18 UTC (permalink / raw)
To: davem, eric.dumazet, paulmck, john.r.fastabend, kaber
Cc: netdev, linux-kernel
In-Reply-To: <cover.1410782841.git.tgraf@suug.ch>
Signed-off-by: Thomas Graf <tgraf@suug.ch>
---
include/linux/spinlock.h | 8 ++++++++
include/linux/spinlock_api_smp.h | 2 ++
include/linux/spinlock_api_up.h | 1 +
kernel/locking/spinlock.c | 8 ++++++++
4 files changed, 19 insertions(+)
diff --git a/include/linux/spinlock.h b/include/linux/spinlock.h
index 3f2867f..481180a 100644
--- a/include/linux/spinlock.h
+++ b/include/linux/spinlock.h
@@ -190,6 +190,8 @@ static inline void do_raw_spin_unlock(raw_spinlock_t *lock) __releases(lock)
#ifdef CONFIG_DEBUG_LOCK_ALLOC
# define raw_spin_lock_nested(lock, subclass) \
_raw_spin_lock_nested(lock, subclass)
+# define raw_spin_lock_bh_nested(lock, subclass) \
+ _raw_spin_lock_bh_nested(lock, subclass)
# define raw_spin_lock_nest_lock(lock, nest_lock) \
do { \
@@ -199,6 +201,7 @@ static inline void do_raw_spin_unlock(raw_spinlock_t *lock) __releases(lock)
#else
# define raw_spin_lock_nested(lock, subclass) _raw_spin_lock(lock)
# define raw_spin_lock_nest_lock(lock, nest_lock) _raw_spin_lock(lock)
+# define raw_spin_lock_bh_nested(lock, subclass) _raw_spin_lock_bh(lock)
#endif
#if defined(CONFIG_SMP) || defined(CONFIG_DEBUG_SPINLOCK)
@@ -318,6 +321,11 @@ do { \
raw_spin_lock_nested(spinlock_check(lock), subclass); \
} while (0)
+#define spin_lock_bh_nested(lock, subclass) \
+do { \
+ raw_spin_lock_bh_nested(spinlock_check(lock), subclass);\
+} while (0)
+
#define spin_lock_nest_lock(lock, nest_lock) \
do { \
raw_spin_lock_nest_lock(spinlock_check(lock), nest_lock); \
diff --git a/include/linux/spinlock_api_smp.h b/include/linux/spinlock_api_smp.h
index 42dfab8..5344268 100644
--- a/include/linux/spinlock_api_smp.h
+++ b/include/linux/spinlock_api_smp.h
@@ -22,6 +22,8 @@ int in_lock_functions(unsigned long addr);
void __lockfunc _raw_spin_lock(raw_spinlock_t *lock) __acquires(lock);
void __lockfunc _raw_spin_lock_nested(raw_spinlock_t *lock, int subclass)
__acquires(lock);
+void __lockfunc _raw_spin_lock_bh_nested(raw_spinlock_t *lock, int subclass)
+ __acquires(lock);
void __lockfunc
_raw_spin_lock_nest_lock(raw_spinlock_t *lock, struct lockdep_map *map)
__acquires(lock);
diff --git a/include/linux/spinlock_api_up.h b/include/linux/spinlock_api_up.h
index d0d1888..d3afef9 100644
--- a/include/linux/spinlock_api_up.h
+++ b/include/linux/spinlock_api_up.h
@@ -57,6 +57,7 @@
#define _raw_spin_lock(lock) __LOCK(lock)
#define _raw_spin_lock_nested(lock, subclass) __LOCK(lock)
+#define _raw_spin_lock_bh_nested(lock, subclass) __LOCK(lock)
#define _raw_read_lock(lock) __LOCK(lock)
#define _raw_write_lock(lock) __LOCK(lock)
#define _raw_spin_lock_bh(lock) __LOCK_BH(lock)
diff --git a/kernel/locking/spinlock.c b/kernel/locking/spinlock.c
index 4b082b5..db3ccb1 100644
--- a/kernel/locking/spinlock.c
+++ b/kernel/locking/spinlock.c
@@ -363,6 +363,14 @@ void __lockfunc _raw_spin_lock_nested(raw_spinlock_t *lock, int subclass)
}
EXPORT_SYMBOL(_raw_spin_lock_nested);
+void __lockfunc _raw_spin_lock_bh_nested(raw_spinlock_t *lock, int subclass)
+{
+ __local_bh_disable_ip(_RET_IP_, SOFTIRQ_LOCK_OFFSET);
+ spin_acquire(&lock->dep_map, subclass, 0, _RET_IP_);
+ LOCK_CONTENDED(lock, do_raw_spin_trylock, do_raw_spin_lock);
+}
+EXPORT_SYMBOL(_raw_spin_lock_bh_nested);
+
unsigned long __lockfunc _raw_spin_lock_irqsave_nested(raw_spinlock_t *lock,
int subclass)
{
--
1.9.3
^ permalink raw reply related
* [PATCH 5/5] rhashtable: Per bucket locks & expansion/shrinking in work queue
From: Thomas Graf @ 2014-09-15 12:18 UTC (permalink / raw)
To: davem, eric.dumazet, paulmck, john.r.fastabend, kaber
Cc: netdev, linux-kernel
In-Reply-To: <cover.1410782841.git.tgraf@suug.ch>
Introduces an array of spinlocks to protect bucket mutations. The number
of spinlocks per CPU is configurable and selected based on the hash of
the bucket. This allows for parallel insertions and removals of entries
which do not share a lock.
The patch also defers expansion and shrinking to a worker queue which
allow insertion and removal from atomic context. Insertions and
deletions may occur in parallel to it and are only held up briefly
while the particular bucket is linked or unzipped.
Mutations of the bucket table pointer is protected by a new mutex, read
access to that pointer for insertion and deletion is RCU protected.
In the event of an expansion or shrinking, the new bucket table allocated
is exposed as a so called future table right away. Lookups, deletions, and
insertions will briefly use both tables. The future table becomes the main
table after an RCU grace period and initial relinking was performed which
guarantees that no new insertions occur on the old table but all entries
can be found.
The side effect of this is that during that RCU grace period, a bucket
traversal using any rht_for_each() variant on the main table will not see
any insertions performed during the RCU grace period which would at that
point land in the future table. The lookup will see them as it searches
both tables if needed.
Signed-off-by: Thomas Graf <tgraf@suug.ch>
---
include/linux/rhashtable.h | 42 +++--
lib/rhashtable.c | 375 +++++++++++++++++++++++++++++++--------------
net/netfilter/nft_hash.c | 59 +++----
net/netlink/af_netlink.c | 5 +-
4 files changed, 330 insertions(+), 151 deletions(-)
diff --git a/include/linux/rhashtable.h b/include/linux/rhashtable.h
index e9cdbda..dc8a447 100644
--- a/include/linux/rhashtable.h
+++ b/include/linux/rhashtable.h
@@ -19,13 +19,23 @@
#define _LINUX_RHASHTABLE_H
#include <linux/list_nulls.h>
+#include <linux/workqueue.h>
struct rhash_head {
struct rhash_head __rcu *next;
};
+/**
+ * struct bucket_table - Table of hash buckets
+ * @size: Number of hash buckets
+ * @locks_mask: Mask to apply before accessing locks[]
+ * @locks: Array of spinlocks protecting individual buckets
+ * @buckets: size * hash buckets
+ */
struct bucket_table {
size_t size;
+ unsigned int locks_mask;
+ spinlock_t *locks;
struct rhash_head __rcu *buckets[];
};
@@ -44,6 +54,7 @@ struct rhashtable;
* @max_shift: Maximum number of shifts while expanding
* @min_shift: Minimum number of shifts while shrinking
* @nulls_base: Base value to generate nulls marker
+ * @locks_mul: Number of bucket locks to allocate per cpu (default: 128)
* @hashfn: Function to hash key
* @obj_hashfn: Function to hash object
* @grow_decision: If defined, may return true if table should expand
@@ -59,6 +70,7 @@ struct rhashtable_params {
size_t max_shift;
size_t min_shift;
int nulls_base;
+ size_t locks_mul;
rht_hashfn_t hashfn;
rht_obj_hashfn_t obj_hashfn;
bool (*grow_decision)(const struct rhashtable *ht,
@@ -71,15 +83,23 @@ struct rhashtable_params {
/**
* struct rhashtable - Hash table handle
* @tbl: Bucket table
+ * @future_tbl: Future table during expansion and shrinking
* @nelems: Number of elements in table
* @shift: Current size (1 << shift)
* @p: Configuration parameters
+ * @run_work: Delayed work executing expansion/shrinking
+ * @mutex: Hold while expansion/shrinking takes places
+ * @being_destroyed: True if table is set to be destroyed
*/
struct rhashtable {
struct bucket_table __rcu *tbl;
+ struct bucket_table __rcu *future_tbl;
size_t nelems;
size_t shift;
struct rhashtable_params p;
+ struct delayed_work run_work;
+ struct mutex mutex;
+ bool being_destroyed;
};
static inline unsigned long rht_marker(const struct rhashtable *ht, u32 hash)
@@ -101,9 +121,16 @@ static inline unsigned long rht_get_nulls_value(const struct rhash_head *ptr)
}
#ifdef CONFIG_PROVE_LOCKING
-int lockdep_rht_mutex_is_held(const struct rhashtable *ht);
+int lockdep_rht_mutex_is_held(struct rhashtable *ht);
+int rht_bucket_lock_is_held(const struct bucket_table *tbl, u32 hash);
#else
-static inline int lockdep_rht_mutex_is_held(const struct rhashtable *ht)
+static inline int lockdep_rht_mutex_is_held(struct rhashtable *ht)
+{
+ return 1;
+}
+
+static inline int lockdep_rht_bucket_is_held(const struct bucket_table *tbl,
+ u32 hash)
{
return 1;
}
@@ -112,12 +139,9 @@ static inline int lockdep_rht_mutex_is_held(const struct rhashtable *ht)
int rhashtable_init(struct rhashtable *ht, struct rhashtable_params *params);
u32 rhashtable_hashfn(const struct rhashtable *ht, const void *key, u32 len);
-u32 rhashtable_obj_hashfn(const struct rhashtable *ht, void *ptr);
void rhashtable_insert(struct rhashtable *ht, struct rhash_head *node);
bool rhashtable_remove(struct rhashtable *ht, struct rhash_head *node);
-void rhashtable_remove_pprev(struct rhashtable *ht, struct rhash_head *obj,
- struct rhash_head __rcu **pprev);
bool rht_grow_above_75(const struct rhashtable *ht, size_t new_size);
bool rht_shrink_below_30(const struct rhashtable *ht, size_t new_size);
@@ -126,10 +150,10 @@ int rhashtable_expand(struct rhashtable *ht);
int rhashtable_shrink(struct rhashtable *ht);
void *rhashtable_lookup(const struct rhashtable *ht, const void *key);
-void *rhashtable_lookup_compare(const struct rhashtable *ht, u32 hash,
+void *rhashtable_lookup_compare(const struct rhashtable *ht, const void *key,
bool (*compare)(void *, void *), void *arg);
-void rhashtable_destroy(const struct rhashtable *ht);
+void rhashtable_destroy(struct rhashtable *ht);
#define rht_dereference(p, ht) \
rcu_dereference_protected(p, lockdep_rht_mutex_is_held(ht))
@@ -138,10 +162,10 @@ void rhashtable_destroy(const struct rhashtable *ht);
rcu_dereference_check(p, lockdep_rht_mutex_is_held(ht))
#define rht_dereference_bucket(p, tbl, hash) \
- rcu_dereference_protected(p, lockdep_rht_mutex_is_held(ht))
+ rcu_dereference_protected(p, lockdep_rht_bucket_is_held(tbl, hash))
#define rht_dereference_bucket_rcu(p, tbl, hash) \
- rcu_dereference_check(p, lockdep_rht_mutex_is_held(ht))
+ rcu_dereference_check(p, lockdep_rht_bucket_is_held(tbl, hash))
#define rht_entry(tpos, pos, member) \
({ tpos = container_of(pos, typeof(*tpos), member); 1; })
diff --git a/lib/rhashtable.c b/lib/rhashtable.c
index d871483..2a6ff3d 100644
--- a/lib/rhashtable.c
+++ b/lib/rhashtable.c
@@ -27,6 +27,7 @@
#define HASH_DEFAULT_SIZE 64UL
#define HASH_MIN_SIZE 4UL
+#define BUCKET_LOCKS_PER_CPU 128UL
/*
* The nulls marker consists of:
@@ -45,14 +46,42 @@
#define HASH_BASE_MIN (1 << (31 - HASH_BASE_BITS))
#define HASH_RESERVED_SPACE (HASH_BASE_BITS + 1)
+/* The bucket lock is selected based on the hash and protects mutations
+ * on a group of hash buckets.
+ *
+ * Important: When holding the bucket lock of both the old and new table
+ * during expansions and shrinking, the old bucket lock must always be
+ * acquired first.
+ */
+static spinlock_t *bucket_lock(const struct bucket_table *tbl, u32 hash)
+{
+ return &tbl->locks[hash & tbl->locks_mask];
+}
+
#define ASSERT_RHT_MUTEX(HT) BUG_ON(!lockdep_rht_mutex_is_held(HT))
#ifdef CONFIG_PROVE_LOCKING
-int lockdep_rht_mutex_is_held(const struct rhashtable *ht)
+int lockdep_rht_mutex_is_held(struct rhashtable *ht)
{
- return ht->p.mutex_is_held();
+#ifdef CONFIG_LOCKDEP
+ return (debug_locks) ? lockdep_is_held(&ht->mutex) : 1;
+#else
+ return 1;
+#endif
}
EXPORT_SYMBOL_GPL(lockdep_rht_mutex_is_held);
+
+int lockdep_rht_bucket_is_held(const struct bucket_table *tbl, u32 hash)
+{
+#ifdef CONFIG_LOCKDEP
+ spinlock_t *lock = bucket_lock(tbl, hash);
+
+ return (debug_locks) ? lockdep_is_held(lock) : 1;
+#else
+ return 1;
+#endif
+}
+EXPORT_SYMBOL_GPL(lockdep_rht_bucket_is_held);
#endif
static void *rht_obj(const struct rhashtable *ht, const struct rhash_head *he)
@@ -85,39 +114,19 @@ static u32 obj_raw_hashfn(const struct rhashtable *ht, const void *ptr)
* @len: length of key
*
* Computes the hash value using the hash function provided in the 'hashfn'
- * of struct rhashtable_params. The returned value is guaranteed to be
- * smaller than the number of buckets in the hash table.
+ * of struct rhashtable_params.
*/
u32 rhashtable_hashfn(const struct rhashtable *ht, const void *key, u32 len)
{
- struct bucket_table *tbl = rht_dereference_rcu(ht->tbl, ht);
u32 hash;
hash = ht->p.hashfn(key, len, ht->p.hash_rnd);
hash >>= HASH_RESERVED_SPACE;
- return rht_bucket_index(hash, tbl);
+ return hash;
}
EXPORT_SYMBOL_GPL(rhashtable_hashfn);
-/**
- * rhashtable_obj_hashfn - compute hash for hashed object
- * @ht: hash table to compuate for
- * @ptr: pointer to hashed object
- *
- * Computes the hash value using the hash function `hashfn` respectively
- * 'obj_hashfn' depending on whether the hash table is set up to work with
- * a fixed length key. The returned value is guaranteed to be smaller than
- * the number of buckets in the hash table.
- */
-u32 rhashtable_obj_hashfn(const struct rhashtable *ht, void *ptr)
-{
- struct bucket_table *tbl = rht_dereference_rcu(ht->tbl, ht);
-
- return rht_bucket_index(obj_raw_hashfn(ht, ptr), tbl);
-}
-EXPORT_SYMBOL_GPL(rhashtable_obj_hashfn);
-
static u32 head_hashfn(const struct rhashtable *ht,
const struct rhash_head *he,
const struct bucket_table *tbl)
@@ -125,6 +134,56 @@ static u32 head_hashfn(const struct rhashtable *ht,
return rht_bucket_index(obj_raw_hashfn(ht, rht_obj(ht, he)), tbl);
}
+static struct rhash_head __rcu **rht_list_tail(struct bucket_table *tbl, u32 n)
+{
+ struct rhash_head __rcu **pprev;
+
+ for (pprev = &tbl->buckets[n];
+ !rht_is_a_nulls(rht_dereference_bucket(*pprev, tbl, n));
+ pprev = &rht_dereference_bucket(*pprev, tbl, n)->next)
+ ;
+
+ return pprev;
+}
+
+static int alloc_bucket_locks(struct rhashtable *ht, struct bucket_table *tbl)
+{
+ unsigned int i, size;
+#if defined(CONFIG_PROVE_LOCKING)
+ unsigned int nr_pcpus = 2;
+#else
+ unsigned int nr_pcpus = num_possible_cpus();
+#endif
+
+ nr_pcpus = min_t(unsigned int, nr_pcpus, 32UL);
+ size = nr_pcpus * ht->p.locks_mul;
+
+ if (sizeof(spinlock_t) != 0) {
+#ifdef CONFIG_NUMA
+ if (size * sizeof(spinlock_t) > PAGE_SIZE)
+ tbl->locks = vmalloc(size * sizeof(spinlock_t));
+ else
+#endif
+ tbl->locks = kmalloc_array(size, sizeof(spinlock_t),
+ GFP_KERNEL);
+ if (!tbl->locks)
+ return -ENOMEM;
+ for (i = 0; i < size; i++)
+ spin_lock_init(&tbl->locks[i]);
+ }
+ tbl->locks_mask = size - 1;
+
+ return 0;
+}
+
+static void bucket_table_free(const struct bucket_table *tbl)
+{
+ if (tbl)
+ kvfree(tbl->locks);
+
+ kvfree(tbl);
+}
+
static struct bucket_table *bucket_table_alloc(struct rhashtable *ht,
size_t nbuckets)
{
@@ -143,16 +202,16 @@ static struct bucket_table *bucket_table_alloc(struct rhashtable *ht,
for (i = 0; i < nbuckets; i++)
INIT_RHT_NULLS_HEAD(tbl->buckets[i], ht, i);
+ if (alloc_bucket_locks(ht, tbl) < 0) {
+ bucket_table_free(tbl);
+ return NULL;
+ }
+
tbl->size = nbuckets;
return tbl;
}
-static void bucket_table_free(const struct bucket_table *tbl)
-{
- kvfree(tbl);
-}
-
/**
* rht_grow_above_75 - returns true if nelems > 0.75 * table-size
* @ht: hash table
@@ -183,8 +242,11 @@ static void hashtable_chain_unzip(const struct rhashtable *ht,
{
struct rhash_head *he, *p;
struct rhash_head __rcu *next;
+ spinlock_t *new_bucket_lock;
u32 hash, new_tbl_idx;
+ BUG_ON(!lockdep_rht_bucket_is_held(old_tbl, n));
+
/* Old bucket empty, no work needed. */
p = rht_get_bucket(old_tbl, n);
if (rht_is_a_nulls(p))
@@ -196,6 +258,10 @@ static void hashtable_chain_unzip(const struct rhashtable *ht,
*/
hash = obj_raw_hashfn(ht, rht_obj(ht, p));
new_tbl_idx = rht_bucket_index(hash, new_tbl);
+
+ new_bucket_lock = bucket_lock(new_tbl, new_tbl_idx);
+ spin_lock_bh_nested(new_bucket_lock, SINGLE_DEPTH_NESTING);
+
rht_for_each_continue(he, p->next, old_tbl, n) {
if (head_hashfn(ht, he, new_tbl) != new_tbl_idx)
break;
@@ -204,7 +270,7 @@ static void hashtable_chain_unzip(const struct rhashtable *ht,
RCU_INIT_POINTER(old_tbl->buckets[n], he);
/* Find the subsequent node which does hash to the same
- * bucket as node P, or NULL if no such node exists.
+ * bucket as node P, or nulls if no such node exists.
*/
INIT_RHT_NULLS_HEAD(next, ht, hash);
if (!rht_is_a_nulls(he)) {
@@ -219,7 +285,9 @@ static void hashtable_chain_unzip(const struct rhashtable *ht,
/* Set p's next pointer to that subsequent node pointer,
* bypassing the nodes which do not hash to p's bucket
*/
- RCU_INIT_POINTER(p->next, next);
+ rcu_assign_pointer(p->next, next);
+
+ spin_unlock_bh(new_bucket_lock);
}
/**
@@ -239,8 +307,9 @@ int rhashtable_expand(struct rhashtable *ht)
{
struct bucket_table *new_tbl, *old_tbl = rht_dereference(ht->tbl, ht);
struct rhash_head *he;
+ spinlock_t *new_bucket_lock, *old_bucket_lock;
unsigned int i, h;
- bool complete;
+ bool complete = false;
ASSERT_RHT_MUTEX(ht);
@@ -253,22 +322,38 @@ int rhashtable_expand(struct rhashtable *ht)
ht->shift++;
- /* For each new bucket, search the corresponding old bucket
- * for the first entry that hashes to the new bucket, and
- * link the new bucket to that entry. Since all the entries
- * which will end up in the new bucket appear in the same
- * old bucket, this constructs an entirely valid new hash
- * table, but with multiple buckets "zipped" together into a
- * single imprecise chain.
+ /* Make insertions go into the new, empty table right away. Deletions
+ * and lookups will be attemped in both tables until we synchronzie.
+ * The synchronize_rcu() guarantees for the new table to be picked up
+ * so no new additions go into the old table while we relink.
+ */
+ rcu_assign_pointer(ht->future_tbl, new_tbl);
+ synchronize_rcu();
+
+ /* For each new bucket, search the corresponding old bucket for the
+ * first entry that hashes to the new bucket, and link the end of
+ * newly formed bucket chain (containing entries added to future
+ * table) to that entry. Since all the entries which will end up in
+ * the new bucket appear in the same old bucket, this constructs an
+ * entirely valid new hash table, but with multiple buckets
+ * "zipped" together into a single imprecise chain.
*/
for (i = 0; i < new_tbl->size; i++) {
h = i & (old_tbl->size - 1);
+ old_bucket_lock = bucket_lock(old_tbl, h);
+ spin_lock_bh(old_bucket_lock);
+
rht_for_each(he, old_tbl, h) {
if (head_hashfn(ht, he, new_tbl) == i) {
- RCU_INIT_POINTER(new_tbl->buckets[i], he);
+ new_bucket_lock = bucket_lock(new_tbl, i);
+ spin_lock_bh_nested(new_bucket_lock,
+ SINGLE_DEPTH_NESTING);
+ rcu_assign_pointer(*rht_list_tail(new_tbl, i), he);
+ spin_unlock_bh(new_bucket_lock);
break;
}
}
+ spin_unlock_bh(old_bucket_lock);
}
/* Publish the new table pointer. Lookups may now traverse
@@ -278,7 +363,7 @@ int rhashtable_expand(struct rhashtable *ht)
rcu_assign_pointer(ht->tbl, new_tbl);
/* Unzip interleaved hash chains */
- do {
+ while (!complete && !ht->being_destroyed) {
/* Wait for readers. All new readers will see the new
* table, and thus no references to the old table will
* remain.
@@ -291,11 +376,18 @@ int rhashtable_expand(struct rhashtable *ht)
*/
complete = true;
for (i = 0; i < old_tbl->size; i++) {
+ spinlock_t *old_bucket_lock;
+
+ old_bucket_lock = bucket_lock(old_tbl, i);
+ spin_lock_bh(old_bucket_lock);
+
hashtable_chain_unzip(ht, new_tbl, old_tbl, i);
- if (!rht_is_a_nulls(old_tbl->buckets[i]))
+ if (!rht_is_a_nulls(rht_get_bucket(old_tbl, i)))
complete = false;
+
+ spin_unlock_bh(old_bucket_lock);
}
- } while (!complete);
+ }
bucket_table_free(old_tbl);
return 0;
@@ -315,7 +407,7 @@ EXPORT_SYMBOL_GPL(rhashtable_expand);
int rhashtable_shrink(struct rhashtable *ht)
{
struct bucket_table *ntbl, *tbl = rht_dereference(ht->tbl, ht);
- struct rhash_head __rcu **pprev;
+ spinlock_t *new_bucket_lock, *old_bucket_lock;
unsigned int i;
ASSERT_RHT_MUTEX(ht);
@@ -327,28 +419,31 @@ int rhashtable_shrink(struct rhashtable *ht)
if (ntbl == NULL)
return -ENOMEM;
- ht->shift--;
+ rcu_assign_pointer(ht->future_tbl, ntbl);
+ synchronize_rcu();
/* Link each bucket in the new table to the first bucket
* in the old table that contains entries which will hash
* to the new bucket.
*/
for (i = 0; i < ntbl->size; i++) {
+ old_bucket_lock = bucket_lock(tbl, rht_bucket_index(i, tbl));
+ new_bucket_lock = bucket_lock(ntbl, i);
+
+ spin_lock_bh(old_bucket_lock);
+ spin_lock_bh_nested(new_bucket_lock, SINGLE_DEPTH_NESTING);
+
ntbl->buckets[i] = tbl->buckets[i];
+ rcu_assign_pointer(*rht_list_tail(ntbl, i),
+ tbl->buckets[i + ntbl->size]);
- /* Link each bucket in the new table to the first bucket
- * in the old table that contains entries which will hash
- * to the new bucket.
- */
- for (pprev = &ntbl->buckets[i];
- !rht_is_a_nulls(rht_dereference_bucket(*pprev, ntbl, i));
- pprev = &rht_dereference_bucket(*pprev, ntbl, i)->next)
- ;
- RCU_INIT_POINTER(*pprev, tbl->buckets[i + ntbl->size]);
+ spin_unlock_bh(new_bucket_lock);
+ spin_unlock_bh(old_bucket_lock);
}
/* Publish the new, valid hash table */
rcu_assign_pointer(ht->tbl, ntbl);
+ ht->shift--;
/* Wait for readers. No new readers will have references to the
* old hash table.
@@ -361,6 +456,23 @@ int rhashtable_shrink(struct rhashtable *ht)
}
EXPORT_SYMBOL_GPL(rhashtable_shrink);
+static void rht_deferred_worker(struct work_struct *work)
+{
+ struct rhashtable *ht;
+ struct bucket_table *tbl;
+
+ ht = container_of(work, struct rhashtable, run_work.work);
+ mutex_lock(&ht->mutex);
+ tbl = rht_dereference(ht->tbl, ht);
+
+ if (ht->p.grow_decision && ht->p.grow_decision(ht, tbl->size))
+ rhashtable_expand(ht);
+ else if (ht->p.shrink_decision && ht->p.shrink_decision(ht, tbl->size))
+ rhashtable_shrink(ht);
+
+ mutex_unlock(&ht->mutex);
+}
+
/**
* rhashtable_insert - insert object into hash hash table
* @ht: hash table
@@ -369,18 +481,23 @@ EXPORT_SYMBOL_GPL(rhashtable_shrink);
* Will automatically grow the table via rhashtable_expand() if the the
* grow_decision function specified at rhashtable_init() returns true.
*
- * The caller must ensure that no concurrent table mutations occur. It is
- * however valid to have concurrent lookups if they are RCU protected.
+ * Will take a per bucket spinlock to protect against mutual mutations
+ * on the same bucket.
*/
void rhashtable_insert(struct rhashtable *ht, struct rhash_head *obj)
{
- struct bucket_table *tbl = rht_dereference(ht->tbl, ht);
+ struct bucket_table *tbl;
+ spinlock_t *lock;
u32 hash, idx;
- ASSERT_RHT_MUTEX(ht);
-
+ rcu_read_lock();
+ tbl = rht_dereference_rcu(ht->future_tbl, ht);
hash = obj_raw_hashfn(ht, rht_obj(ht, obj));
idx = rht_bucket_index(hash, tbl);
+
+ lock = bucket_lock(tbl, idx);
+ spin_lock_bh(lock);
+
if (rht_is_a_nulls(rht_get_bucket(tbl, idx)))
INIT_RHT_NULLS_HEAD(obj->next, ht, hash);
else
@@ -388,36 +505,14 @@ void rhashtable_insert(struct rhashtable *ht, struct rhash_head *obj)
rcu_assign_pointer(tbl->buckets[idx], obj);
ht->nelems++;
- if (ht->p.grow_decision && ht->p.grow_decision(ht, tbl->size))
- rhashtable_expand(ht);
-}
-EXPORT_SYMBOL_GPL(rhashtable_insert);
-
-/**
- * rhashtable_remove_pprev - remove object from hash table given previous element
- * @ht: hash table
- * @obj: pointer to hash head inside object
- * @pprev: pointer to previous element
- *
- * Identical to rhashtable_remove() but caller is alreayd aware of the element
- * in front of the element to be deleted. This is in particular useful for
- * deletion when combined with walking or lookup.
- */
-void rhashtable_remove_pprev(struct rhashtable *ht, struct rhash_head *obj,
- struct rhash_head __rcu **pprev)
-{
- struct bucket_table *tbl = rht_dereference(ht->tbl, ht);
-
- ASSERT_RHT_MUTEX(ht);
+ spin_unlock_bh(lock);
- RCU_INIT_POINTER(*pprev, obj->next);
- ht->nelems--;
+ if (ht->p.grow_decision && ht->p.grow_decision(ht, tbl->size))
+ schedule_delayed_work(&ht->run_work, 0);
- if (ht->p.shrink_decision &&
- ht->p.shrink_decision(ht, tbl->size))
- rhashtable_shrink(ht);
+ rcu_read_unlock();
}
-EXPORT_SYMBOL_GPL(rhashtable_remove_pprev);
+EXPORT_SYMBOL_GPL(rhashtable_insert);
/**
* rhashtable_remove - remove object from hash table
@@ -436,14 +531,20 @@ EXPORT_SYMBOL_GPL(rhashtable_remove_pprev);
*/
bool rhashtable_remove(struct rhashtable *ht, struct rhash_head *obj)
{
- struct bucket_table *tbl = rht_dereference(ht->tbl, ht);
- struct rhash_head __rcu **pprev;
+ struct bucket_table *tbl;
struct rhash_head *he;
+ struct rhash_head __rcu **pprev;
+ spinlock_t *lock;
u32 idx;
- ASSERT_RHT_MUTEX(ht);
-
+ rcu_read_lock();
+ tbl = rht_dereference_rcu(ht->future_tbl, ht);
idx = head_hashfn(ht, obj, tbl);
+
+ lock = bucket_lock(tbl, idx);
+ spin_lock_bh(lock);
+
+restart:
pprev = &tbl->buckets[idx];
rht_for_each(he, tbl, idx) {
if (he != obj) {
@@ -451,10 +552,34 @@ bool rhashtable_remove(struct rhashtable *ht, struct rhash_head *obj)
continue;
}
- rhashtable_remove_pprev(ht, he, pprev);
+ rcu_assign_pointer(*pprev, obj->next);
+ ht->nelems--;
+
+ spin_unlock_bh(lock);
+
+ if (ht->p.shrink_decision &&
+ ht->p.shrink_decision(ht, tbl->size))
+ schedule_delayed_work(&ht->run_work, 0);
+
+ rcu_read_unlock();
+
return true;
}
+ if (tbl != rht_dereference_rcu(ht->tbl, ht)) {
+ spin_unlock_bh(lock);
+
+ tbl = rht_dereference_rcu(ht->tbl, ht);
+ idx = head_hashfn(ht, obj, tbl);
+
+ lock = bucket_lock(tbl, idx);
+ spin_lock_bh(lock);
+ goto restart;
+ }
+
+ spin_unlock_bh(lock);
+ rcu_read_unlock();
+
return false;
}
EXPORT_SYMBOL_GPL(rhashtable_remove);
@@ -471,24 +596,32 @@ EXPORT_SYMBOL_GPL(rhashtable_remove);
* paramter set). It will BUG() if used inappropriately.
*
* Lookups may occur in parallel with hash mutations as long as the lookup is
- * guarded by rcu_read_lock(). The caller must take care of this.
+ * guarded by rcu_read_lock(). Otherwise the ht->mutex must be held.
*/
void *rhashtable_lookup(const struct rhashtable *ht, const void *key)
{
- const struct bucket_table *tbl = rht_dereference_rcu(ht->tbl, ht);
+ const struct bucket_table *tbl, *old_tbl;
struct rhash_head *he;
u32 h;
BUG_ON(!ht->p.key_len);
+ old_tbl = rht_dereference_rcu(ht->tbl, ht);
+ tbl = rht_dereference_rcu(ht->future_tbl, ht);
h = rhashtable_hashfn(ht, key, ht->p.key_len);
- rht_for_each_rcu(he, tbl, h) {
+restart:
+ rht_for_each_rcu(he, tbl, rht_bucket_index(h, tbl)) {
if (memcmp(rht_obj(ht, he) + ht->p.key_offset, key,
ht->p.key_len))
continue;
return rht_obj(ht, he);
}
+ if (unlikely(tbl != old_tbl)) {
+ tbl = old_tbl;
+ goto restart;
+ }
+
return NULL;
}
EXPORT_SYMBOL_GPL(rhashtable_lookup);
@@ -496,7 +629,7 @@ EXPORT_SYMBOL_GPL(rhashtable_lookup);
/**
* rhashtable_lookup_compare - search hash table with compare function
* @ht: hash table
- * @hash: hash value of desired entry
+ * @key: the pointer to the key
* @compare: compare function, must return true on match
* @arg: argument passed on to compare function
*
@@ -508,21 +641,28 @@ EXPORT_SYMBOL_GPL(rhashtable_lookup);
*
* Returns the first entry on which the compare function returned true.
*/
-void *rhashtable_lookup_compare(const struct rhashtable *ht, u32 hash,
+void *rhashtable_lookup_compare(const struct rhashtable *ht, const void *key,
bool (*compare)(void *, void *), void *arg)
{
- const struct bucket_table *tbl = rht_dereference_rcu(ht->tbl, ht);
+ const struct bucket_table *tbl, *old_tbl;
struct rhash_head *he;
+ u32 h;
- if (unlikely(hash >= tbl->size))
- return NULL;
-
- rht_for_each_rcu(he, tbl, hash) {
+ old_tbl = rht_dereference_rcu(ht->tbl, ht);
+ tbl = rht_dereference_rcu(ht->future_tbl, ht);
+ h = rhashtable_hashfn(ht, key, ht->p.key_len);
+restart:
+ rht_for_each_rcu(he, tbl, rht_bucket_index(h, tbl)) {
if (!compare(rht_obj(ht, he), arg))
continue;
return (void *) he - ht->p.head_offset;
}
+ if (unlikely(tbl != old_tbl)) {
+ tbl = old_tbl;
+ goto restart;
+ }
+
return NULL;
}
EXPORT_SYMBOL_GPL(rhashtable_lookup_compare);
@@ -554,7 +694,6 @@ static size_t rounded_hashtable_size(struct rhashtable_params *params)
* .key_offset = offsetof(struct test_obj, key),
* .key_len = sizeof(int),
* .hashfn = arch_fast_hash,
- * .mutex_is_held = &my_mutex_is_held,
* };
*
* Configuration Example 2: Variable length keys
@@ -574,7 +713,6 @@ static size_t rounded_hashtable_size(struct rhashtable_params *params)
* .head_offset = offsetof(struct test_obj, node),
* .hashfn = arch_fast_hash,
* .obj_hashfn = my_hash_fn,
- * .mutex_is_held = &my_mutex_is_held,
* };
*/
int rhashtable_init(struct rhashtable *ht, struct rhashtable_params *params)
@@ -599,6 +737,12 @@ int rhashtable_init(struct rhashtable *ht, struct rhashtable_params *params)
memset(ht, 0, sizeof(*ht));
memcpy(&ht->p, params, sizeof(*params));
+ mutex_init(&ht->mutex);
+
+ if (params->locks_mul)
+ ht->p.locks_mul = roundup_pow_of_two(params->locks_mul);
+ else
+ ht->p.locks_mul = BUCKET_LOCKS_PER_CPU;
tbl = bucket_table_alloc(ht, size);
if (tbl == NULL)
@@ -606,10 +750,14 @@ int rhashtable_init(struct rhashtable *ht, struct rhashtable_params *params)
ht->shift = ilog2(tbl->size);
RCU_INIT_POINTER(ht->tbl, tbl);
+ RCU_INIT_POINTER(ht->future_tbl, tbl);
if (!ht->p.hash_rnd)
get_random_bytes(&ht->p.hash_rnd, sizeof(ht->p.hash_rnd));
+ if (ht->p.grow_decision || ht->p.shrink_decision)
+ INIT_DEFERRABLE_WORK(&ht->run_work, rht_deferred_worker);
+
return 0;
}
EXPORT_SYMBOL_GPL(rhashtable_init);
@@ -620,11 +768,16 @@ EXPORT_SYMBOL_GPL(rhashtable_init);
*
* Frees the bucket array.
*/
-void rhashtable_destroy(const struct rhashtable *ht)
+void rhashtable_destroy(struct rhashtable *ht)
{
- const struct bucket_table *tbl = rht_dereference(ht->tbl, ht);
+ ht->being_destroyed = true;
- bucket_table_free(tbl);
+ mutex_lock(&ht->mutex);
+
+ cancel_delayed_work(&ht->run_work);
+ bucket_table_free(rht_dereference(ht->tbl, ht));
+
+ mutex_unlock(&ht->mutex);
}
EXPORT_SYMBOL_GPL(rhashtable_destroy);
@@ -639,11 +792,6 @@ EXPORT_SYMBOL_GPL(rhashtable_destroy);
#define TEST_PTR ((void *) 0xdeadbeef)
#define TEST_NEXPANDS 4
-static int test_mutex_is_held(void)
-{
- return 1;
-}
-
struct test_obj {
void *ptr;
int value;
@@ -755,7 +903,9 @@ static int __init test_rhashtable(struct rhashtable *ht)
for (i = 0; i < TEST_NEXPANDS; i++) {
pr_info(" Table expansion iteration %u...\n", i);
+ mutex_lock(&ht->mutex);
rhashtable_expand(ht);
+ mutex_unlock(&ht->mutex);
rcu_read_lock();
pr_info(" Verifying lookups...\n");
@@ -765,7 +915,9 @@ static int __init test_rhashtable(struct rhashtable *ht)
for (i = 0; i < TEST_NEXPANDS; i++) {
pr_info(" Table shrinkage iteration %u...\n", i);
+ mutex_lock(&ht->mutex);
rhashtable_shrink(ht);
+ mutex_unlock(&ht->mutex);
rcu_read_lock();
pr_info(" Verifying lookups...\n");
@@ -804,7 +956,6 @@ static int __init test_rht_init(void)
.key_offset = offsetof(struct test_obj, value),
.key_len = sizeof(int),
.hashfn = arch_fast_hash,
- .mutex_is_held = &test_mutex_is_held,
.grow_decision = rht_grow_above_75,
.shrink_decision = rht_shrink_below_30,
};
diff --git a/net/netfilter/nft_hash.c b/net/netfilter/nft_hash.c
index 68b654b..436f77b 100644
--- a/net/netfilter/nft_hash.c
+++ b/net/netfilter/nft_hash.c
@@ -83,41 +83,48 @@ static void nft_hash_remove(const struct nft_set *set,
const struct nft_set_elem *elem)
{
struct rhashtable *priv = nft_set_priv(set);
- struct rhash_head *he, __rcu **pprev;
-
- pprev = elem->cookie;
- he = rht_dereference((*pprev), priv);
-
- rhashtable_remove_pprev(priv, he, pprev);
+ struct rhash_head *he = elem->cookie;
+ rhashtable_remove(priv, he);
synchronize_rcu();
kfree(he);
}
+struct nft_compare_arg {
+ const struct nft_set *set;
+ struct nft_set_elem *elem;
+};
+
+static bool nft_hash_compare(void *ptr, void *arg)
+{
+ struct nft_hash_elem *he = ptr;
+ struct nft_compare_arg *x = arg;
+
+ if (!nft_data_cmp(&he->key, &x->elem->key, x->set->klen)) {
+ x->elem->cookie = &he->node;
+ x->elem->flags = 0;
+ if (x->set->flags & NFT_SET_MAP)
+ nft_data_copy(&x->elem->data, he->data);
+
+ return true;
+ }
+
+ return false;
+}
+
static int nft_hash_get(const struct nft_set *set, struct nft_set_elem *elem)
{
const struct rhashtable *priv = nft_set_priv(set);
- const struct bucket_table *tbl = rht_dereference_rcu(priv->tbl, priv);
- struct rhash_head __rcu * const *pprev;
- struct rhash_head *pos;
- struct nft_hash_elem *he;
- u32 h;
-
- h = rhashtable_hashfn(priv, &elem->key, set->klen);
- pprev = &tbl->buckets[h];
- rht_for_each_entry_rcu(he, pos, tbl, h, node) {
- if (nft_data_cmp(&he->key, &elem->key, set->klen)) {
- pprev = &he->node.next;
- continue;
- }
+ struct nft_compare_arg arg = {
+ .set = set,
+ .elem = elem,
+ };
- elem->cookie = (void *)pprev;
- elem->flags = 0;
- if (set->flags & NFT_SET_MAP)
- nft_data_copy(&elem->data, he->data);
+ if (rhashtable_lookup_compare(priv, &elem->key,
+ &nft_hash_compare, &arg))
return 0;
- }
- return -ENOENT;
+ else
+ return -ENOENT;
}
static void nft_hash_walk(const struct nft_ctx *ctx, const struct nft_set *set,
@@ -182,7 +189,7 @@ static int nft_hash_init(const struct nft_set *set,
static void nft_hash_destroy(const struct nft_set *set)
{
- const struct rhashtable *priv = nft_set_priv(set);
+ struct rhashtable *priv = nft_set_priv(set);
const struct bucket_table *tbl;
struct nft_hash_elem *he;
struct rhash_head *pos, *next;
diff --git a/net/netlink/af_netlink.c b/net/netlink/af_netlink.c
index 98e5b58..4e02fa2 100644
--- a/net/netlink/af_netlink.c
+++ b/net/netlink/af_netlink.c
@@ -1015,11 +1015,8 @@ static struct sock *__netlink_lookup(struct netlink_table *table, u32 portid,
.net = net,
.portid = portid,
};
- u32 hash;
- hash = rhashtable_hashfn(&table->hash, &portid, sizeof(portid));
-
- return rhashtable_lookup_compare(&table->hash, hash,
+ return rhashtable_lookup_compare(&table->hash, &portid,
&netlink_compare, &arg);
}
--
1.9.3
^ permalink raw reply related
* [PATCH 1/5] rhashtable: Remove gfp_flags from insert and remove functions
From: Thomas Graf @ 2014-09-15 12:18 UTC (permalink / raw)
To: davem, eric.dumazet, paulmck, john.r.fastabend, kaber
Cc: netdev, linux-kernel
In-Reply-To: <cover.1410782841.git.tgraf@suug.ch>
As the expansion/shrinking is moved to a worker thread, no allocations
will be performed anymore.
Signed-off-by: Thomas Graf <tgraf@suug.ch>
---
include/linux/rhashtable.h | 10 +++++-----
lib/rhashtable.c | 41 +++++++++++++++++------------------------
net/netfilter/nft_hash.c | 4 ++--
net/netlink/af_netlink.c | 4 ++--
4 files changed, 26 insertions(+), 33 deletions(-)
diff --git a/include/linux/rhashtable.h b/include/linux/rhashtable.h
index fb298e9d..942fa44 100644
--- a/include/linux/rhashtable.h
+++ b/include/linux/rhashtable.h
@@ -96,16 +96,16 @@ int rhashtable_init(struct rhashtable *ht, struct rhashtable_params *params);
u32 rhashtable_hashfn(const struct rhashtable *ht, const void *key, u32 len);
u32 rhashtable_obj_hashfn(const struct rhashtable *ht, void *ptr);
-void rhashtable_insert(struct rhashtable *ht, struct rhash_head *node, gfp_t);
-bool rhashtable_remove(struct rhashtable *ht, struct rhash_head *node, gfp_t);
+void rhashtable_insert(struct rhashtable *ht, struct rhash_head *node);
+bool rhashtable_remove(struct rhashtable *ht, struct rhash_head *node);
void rhashtable_remove_pprev(struct rhashtable *ht, struct rhash_head *obj,
- struct rhash_head __rcu **pprev, gfp_t flags);
+ struct rhash_head __rcu **pprev);
bool rht_grow_above_75(const struct rhashtable *ht, size_t new_size);
bool rht_shrink_below_30(const struct rhashtable *ht, size_t new_size);
-int rhashtable_expand(struct rhashtable *ht, gfp_t flags);
-int rhashtable_shrink(struct rhashtable *ht, gfp_t flags);
+int rhashtable_expand(struct rhashtable *ht);
+int rhashtable_shrink(struct rhashtable *ht);
void *rhashtable_lookup(const struct rhashtable *ht, const void *key);
void *rhashtable_lookup_compare(const struct rhashtable *ht, u32 hash,
diff --git a/lib/rhashtable.c b/lib/rhashtable.c
index 8dfec3f..c133d82 100644
--- a/lib/rhashtable.c
+++ b/lib/rhashtable.c
@@ -108,13 +108,13 @@ static u32 head_hashfn(const struct rhashtable *ht,
return obj_hashfn(ht, rht_obj(ht, he), hsize);
}
-static struct bucket_table *bucket_table_alloc(size_t nbuckets, gfp_t flags)
+static struct bucket_table *bucket_table_alloc(size_t nbuckets)
{
struct bucket_table *tbl;
size_t size;
size = sizeof(*tbl) + nbuckets * sizeof(tbl->buckets[0]);
- tbl = kzalloc(size, flags);
+ tbl = kzalloc(size, GFP_KERNEL);
if (tbl == NULL)
tbl = vzalloc(size);
@@ -201,7 +201,6 @@ static void hashtable_chain_unzip(const struct rhashtable *ht,
/**
* rhashtable_expand - Expand hash table while allowing concurrent lookups
* @ht: the hash table to expand
- * @flags: allocation flags
*
* A secondary bucket array is allocated and the hash entries are migrated
* while keeping them on both lists until the end of the RCU grace period.
@@ -212,7 +211,7 @@ static void hashtable_chain_unzip(const struct rhashtable *ht,
* The caller must ensure that no concurrent table mutations take place.
* It is however valid to have concurrent lookups if they are RCU protected.
*/
-int rhashtable_expand(struct rhashtable *ht, gfp_t flags)
+int rhashtable_expand(struct rhashtable *ht)
{
struct bucket_table *new_tbl, *old_tbl = rht_dereference(ht->tbl, ht);
struct rhash_head *he;
@@ -224,7 +223,7 @@ int rhashtable_expand(struct rhashtable *ht, gfp_t flags)
if (ht->p.max_shift && ht->shift >= ht->p.max_shift)
return 0;
- new_tbl = bucket_table_alloc(old_tbl->size * 2, flags);
+ new_tbl = bucket_table_alloc(old_tbl->size * 2);
if (new_tbl == NULL)
return -ENOMEM;
@@ -282,7 +281,6 @@ EXPORT_SYMBOL_GPL(rhashtable_expand);
/**
* rhashtable_shrink - Shrink hash table while allowing concurrent lookups
* @ht: the hash table to shrink
- * @flags: allocation flags
*
* This function may only be called in a context where it is safe to call
* synchronize_rcu(), e.g. not within a rcu_read_lock() section.
@@ -290,7 +288,7 @@ EXPORT_SYMBOL_GPL(rhashtable_expand);
* The caller must ensure that no concurrent table mutations take place.
* It is however valid to have concurrent lookups if they are RCU protected.
*/
-int rhashtable_shrink(struct rhashtable *ht, gfp_t flags)
+int rhashtable_shrink(struct rhashtable *ht)
{
struct bucket_table *ntbl, *tbl = rht_dereference(ht->tbl, ht);
struct rhash_head __rcu **pprev;
@@ -301,7 +299,7 @@ int rhashtable_shrink(struct rhashtable *ht, gfp_t flags)
if (ht->shift <= ht->p.min_shift)
return 0;
- ntbl = bucket_table_alloc(tbl->size / 2, flags);
+ ntbl = bucket_table_alloc(tbl->size / 2);
if (ntbl == NULL)
return -ENOMEM;
@@ -342,7 +340,6 @@ EXPORT_SYMBOL_GPL(rhashtable_shrink);
* rhashtable_insert - insert object into hash hash table
* @ht: hash table
* @obj: pointer to hash head inside object
- * @flags: allocation flags (table expansion)
*
* Will automatically grow the table via rhashtable_expand() if the the
* grow_decision function specified at rhashtable_init() returns true.
@@ -350,8 +347,7 @@ EXPORT_SYMBOL_GPL(rhashtable_shrink);
* The caller must ensure that no concurrent table mutations occur. It is
* however valid to have concurrent lookups if they are RCU protected.
*/
-void rhashtable_insert(struct rhashtable *ht, struct rhash_head *obj,
- gfp_t flags)
+void rhashtable_insert(struct rhashtable *ht, struct rhash_head *obj)
{
struct bucket_table *tbl = rht_dereference(ht->tbl, ht);
u32 hash;
@@ -364,7 +360,7 @@ void rhashtable_insert(struct rhashtable *ht, struct rhash_head *obj,
ht->nelems++;
if (ht->p.grow_decision && ht->p.grow_decision(ht, tbl->size))
- rhashtable_expand(ht, flags);
+ rhashtable_expand(ht);
}
EXPORT_SYMBOL_GPL(rhashtable_insert);
@@ -373,14 +369,13 @@ EXPORT_SYMBOL_GPL(rhashtable_insert);
* @ht: hash table
* @obj: pointer to hash head inside object
* @pprev: pointer to previous element
- * @flags: allocation flags (table expansion)
*
* Identical to rhashtable_remove() but caller is alreayd aware of the element
* in front of the element to be deleted. This is in particular useful for
* deletion when combined with walking or lookup.
*/
void rhashtable_remove_pprev(struct rhashtable *ht, struct rhash_head *obj,
- struct rhash_head __rcu **pprev, gfp_t flags)
+ struct rhash_head __rcu **pprev)
{
struct bucket_table *tbl = rht_dereference(ht->tbl, ht);
@@ -391,7 +386,7 @@ void rhashtable_remove_pprev(struct rhashtable *ht, struct rhash_head *obj,
if (ht->p.shrink_decision &&
ht->p.shrink_decision(ht, tbl->size))
- rhashtable_shrink(ht, flags);
+ rhashtable_shrink(ht);
}
EXPORT_SYMBOL_GPL(rhashtable_remove_pprev);
@@ -399,7 +394,6 @@ EXPORT_SYMBOL_GPL(rhashtable_remove_pprev);
* rhashtable_remove - remove object from hash table
* @ht: hash table
* @obj: pointer to hash head inside object
- * @flags: allocation flags (table expansion)
*
* Since the hash chain is single linked, the removal operation needs to
* walk the bucket chain upon removal. The removal operation is thus
@@ -411,8 +405,7 @@ EXPORT_SYMBOL_GPL(rhashtable_remove_pprev);
* The caller must ensure that no concurrent table mutations occur. It is
* however valid to have concurrent lookups if they are RCU protected.
*/
-bool rhashtable_remove(struct rhashtable *ht, struct rhash_head *obj,
- gfp_t flags)
+bool rhashtable_remove(struct rhashtable *ht, struct rhash_head *obj)
{
struct bucket_table *tbl = rht_dereference(ht->tbl, ht);
struct rhash_head __rcu **pprev;
@@ -430,7 +423,7 @@ bool rhashtable_remove(struct rhashtable *ht, struct rhash_head *obj,
continue;
}
- rhashtable_remove_pprev(ht, he, pprev, flags);
+ rhashtable_remove_pprev(ht, he, pprev);
return true;
}
@@ -573,7 +566,7 @@ int rhashtable_init(struct rhashtable *ht, struct rhashtable_params *params)
if (params->nelem_hint)
size = rounded_hashtable_size(params);
- tbl = bucket_table_alloc(size, GFP_KERNEL);
+ tbl = bucket_table_alloc(size);
if (tbl == NULL)
return -ENOMEM;
@@ -708,7 +701,7 @@ static int __init test_rhashtable(struct rhashtable *ht)
obj->ptr = TEST_PTR;
obj->value = i * 2;
- rhashtable_insert(ht, &obj->node, GFP_KERNEL);
+ rhashtable_insert(ht, &obj->node);
}
rcu_read_lock();
@@ -719,7 +712,7 @@ static int __init test_rhashtable(struct rhashtable *ht)
for (i = 0; i < TEST_NEXPANDS; i++) {
pr_info(" Table expansion iteration %u...\n", i);
- rhashtable_expand(ht, GFP_KERNEL);
+ rhashtable_expand(ht);
rcu_read_lock();
pr_info(" Verifying lookups...\n");
@@ -729,7 +722,7 @@ static int __init test_rhashtable(struct rhashtable *ht)
for (i = 0; i < TEST_NEXPANDS; i++) {
pr_info(" Table shrinkage iteration %u...\n", i);
- rhashtable_shrink(ht, GFP_KERNEL);
+ rhashtable_shrink(ht);
rcu_read_lock();
pr_info(" Verifying lookups...\n");
@@ -744,7 +737,7 @@ static int __init test_rhashtable(struct rhashtable *ht)
obj = rhashtable_lookup(ht, &key);
BUG_ON(!obj);
- rhashtable_remove(ht, &obj->node, GFP_KERNEL);
+ rhashtable_remove(ht, &obj->node);
kfree(obj);
}
diff --git a/net/netfilter/nft_hash.c b/net/netfilter/nft_hash.c
index 28fb8f3..b52873c 100644
--- a/net/netfilter/nft_hash.c
+++ b/net/netfilter/nft_hash.c
@@ -65,7 +65,7 @@ static int nft_hash_insert(const struct nft_set *set,
if (set->flags & NFT_SET_MAP)
nft_data_copy(he->data, &elem->data);
- rhashtable_insert(priv, &he->node, GFP_KERNEL);
+ rhashtable_insert(priv, &he->node);
return 0;
}
@@ -88,7 +88,7 @@ static void nft_hash_remove(const struct nft_set *set,
pprev = elem->cookie;
he = rht_dereference((*pprev), priv);
- rhashtable_remove_pprev(priv, he, pprev, GFP_KERNEL);
+ rhashtable_remove_pprev(priv, he, pprev);
synchronize_rcu();
kfree(he);
diff --git a/net/netlink/af_netlink.c b/net/netlink/af_netlink.c
index c416725..a1e6104 100644
--- a/net/netlink/af_netlink.c
+++ b/net/netlink/af_netlink.c
@@ -1082,7 +1082,7 @@ static int netlink_insert(struct sock *sk, struct net *net, u32 portid)
nlk_sk(sk)->portid = portid;
sock_hold(sk);
- rhashtable_insert(&table->hash, &nlk_sk(sk)->node, GFP_KERNEL);
+ rhashtable_insert(&table->hash, &nlk_sk(sk)->node);
err = 0;
err:
mutex_unlock(&nl_sk_hash_lock);
@@ -1095,7 +1095,7 @@ static void netlink_remove(struct sock *sk)
mutex_lock(&nl_sk_hash_lock);
table = &nl_table[sk->sk_protocol];
- if (rhashtable_remove(&table->hash, &nlk_sk(sk)->node, GFP_KERNEL)) {
+ if (rhashtable_remove(&table->hash, &nlk_sk(sk)->node)) {
WARN_ON(atomic_read(&sk->sk_refcnt) == 1);
__sock_put(sk);
}
--
1.9.3
^ permalink raw reply related
* [PATCH 0/5 RFC net-next] rhashtable: Parallel atomic insert/deletion & deferred expansion
From: Thomas Graf @ 2014-09-15 12:17 UTC (permalink / raw)
To: davem, eric.dumazet, paulmck, john.r.fastabend, kaber
Cc: netdev, linux-kernel
This is still work in progress but I wanted to share this as soon as the
direction where this is heading to becomes clear.
* Patch 1: Remove gfp_t awareness
* Patch 2: Enhance selftest to catch traversal miscount
* Patch 3: Nulls marker convertion
* Patch 4: Add spin_lock_bh_nested()
* Patch 5: Per bucket spinlock, deferred expansion/shrinking
Paul, Eric, John, Dave: Let me know how far off this is from what you had
in mind.
Opens:
* I'm in particular interested in the feasibility of the side effect
regarding recent insertion visibility in bucket traversals (see below).
I think current users of rhashtable can live with it but it might be
worth to offer a synchronous expansion alternative at some point.
If anybody has a better alternative I'm definitely interested to hear
about it.
* Convert inet hashtable over to this and see if it is missing anything.
* Own worker thread as we might eventually sleep for multiple grace
periods?
* Yes, some of the rcu_assign_pointer() can probably be replaced with
RCU_INIT_POINTER().
Converts the single linked list to us a nulls marker. Reserves 4 bits to
distinguish list membership. Stores a full 27 bit hash as the default
marker. This allows the marker to stay valid even if the stable expands
or shrinks.
Introduces an array of spinlocks to protect bucket mutations. The number
of spinlocks per CPU is configurable and selected based on the hash of
the bucket. This allows for parallel insertions and removals of entries
which do not share a lock.
The patch also defers expansion and shrinking to a worker queue which
allow insertion and removal from atomic context. Insertions and
deletions may occur in parallel to it and are only held up briefly
while the particular bucket is linked or unzipped.
Mutations of the bucket table pointer is protected by a new mutex taken
during expansion and shrinking, read access to that pointer for insertion
and deletion is RCU protected.
In the event of an expansion or shrinking, the new bucket table allocated
is exposed as a so called future table right away. Lookups, deletions, and
insertions will briefly use both tables. The future table becomes the main
table after an RCU grace period and initial relinking was performed which
guarantees that no new insertions occur on the old table and all entries
can be found.
The side effect of this is that during that RCU grace period, a bucket
traversal using any rht_for_each() variant on the main table will not see
any insertions performed during the RCU grace period which would at that
point land in the future table. The lookup will see them as it searches
both tables if needed.
Thomas Graf (5):
rhashtable: Remove gfp_flags from insert and remove functions
rhashtable: Check for count misatch in selftest
rhashtable: Convert to nulls list
spinlock: Add spin_lock_bh_nested()
rhashtable: Per bucket locks & expansion/shrinking in work queue
include/linux/list_nulls.h | 3 +-
include/linux/rhashtable.h | 241 +++++++++++------
include/linux/spinlock.h | 8 +
include/linux/spinlock_api_smp.h | 2 +
include/linux/spinlock_api_up.h | 1 +
kernel/locking/spinlock.c | 8 +
lib/rhashtable.c | 539 ++++++++++++++++++++++++++-------------
net/netfilter/nft_hash.c | 69 ++---
net/netlink/af_netlink.c | 18 +-
net/netlink/diag.c | 4 +-
10 files changed, 604 insertions(+), 289 deletions(-)
--
1.9.3
^ permalink raw reply
* Re: [BUG REPORT] Unencrypted packets after SNAT, allthough IPSEC-Policies are present
From: Steffen Klassert @ 2014-09-15 12:04 UTC (permalink / raw)
To: Konstantinos Kolelis
Cc: netdev, davem, kuznet, jmorris, yoshfuji, kaber, herbert
In-Reply-To: <20140915080941.GP6390@secunet.com>
On Mon, Sep 15, 2014 at 10:09:41AM +0200, Steffen Klassert wrote:
>
> diff --git a/net/xfrm/xfrm_policy.c b/net/xfrm/xfrm_policy.c
> index beeed60..e041822 100644
> --- a/net/xfrm/xfrm_policy.c
> +++ b/net/xfrm/xfrm_policy.c
> @@ -2138,7 +2138,8 @@ struct dst_entry *xfrm_lookup(struct net *net, struct dst_entry *dst_orig,
> xfrm_pols_put(pols, drop_pols);
> XFRM_INC_STATS(net, LINUX_MIB_XFRMOUTNOSTATES);
>
> - return make_blackhole(net, family, dst_orig);
> + err = -EREMOTE;
> + goto error;
We must return here, otherwise we drop some refcounts too much.
I'll send an updated patch tomorrow.
^ permalink raw reply
* Re: [PATCH] iproute: update dsfield file values
From: Jesper Dangaard Brouer @ 2014-09-15 11:52 UTC (permalink / raw)
To: Eric Dumazet; +Cc: Stephen Hemminger, netdev, brouer
In-Reply-To: <1410781699.7106.153.camel@edumazet-glaptop2.roam.corp.google.com>
On Mon, 15 Sep 2014 04:48:19 -0700
Eric Dumazet <eric.dumazet@gmail.com> wrote:
> On Mon, 2014-09-15 at 13:06 +0200, Jesper Dangaard Brouer wrote:
>
> > Is it a bug, that we put DSCP's Expedited Forwarding (EF) into the
> > best-effort priority band(1) in pfifo_fast???
> >
>
> With 3 bands pfifo_fast, its hard to please everyone.
>
> Using 5 or 6 bands would be more generic ;)
We can easily fix it for EF (which should have gone into band 0).
But we should likely just ignore it for AFxx.
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Sr. Network Kernel Developer at Red Hat
Author of http://www.iptv-analyzer.org
LinkedIn: http://www.linkedin.com/in/brouer
^ permalink raw reply
* Re: [PATCH] iproute: update dsfield file values
From: Eric Dumazet @ 2014-09-15 11:48 UTC (permalink / raw)
To: Jesper Dangaard Brouer; +Cc: Stephen Hemminger, netdev
In-Reply-To: <20140915130608.5d238cb1@redhat.com>
On Mon, 2014-09-15 at 13:06 +0200, Jesper Dangaard Brouer wrote:
> Is it a bug, that we put DSCP's Expedited Forwarding (EF) into the
> best-effort priority band(1) in pfifo_fast???
>
With 3 bands pfifo_fast, its hard to please everyone.
Using 5 or 6 bands would be more generic ;)
^ permalink raw reply
* [PATCH v2 net-next 3/3] tcp: do not copy headers in tcp_collapse()
From: Eric Dumazet @ 2014-09-15 11:19 UTC (permalink / raw)
To: David S. Miller; +Cc: netdev, Yuchung Cheng, Neal Cardwell, Eric Dumazet
In-Reply-To: <1410779993-31464-1-git-send-email-edumazet@google.com>
tcp_collapse() wants to shrink skb so that the overhead is minimal.
Now we store tcp flags into TCP_SKB_CB(skb)->tcp_flags, we no longer
need to keep around full headers.
Whole available space is dedicated to the payload.
Signed-off-by: Eric Dumazet <edumazet@google.com>
---
net/ipv4/tcp_input.c | 17 ++---------------
1 file changed, 2 insertions(+), 15 deletions(-)
diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c
index 228bf0c5ff19..ea92f23ffaf1 100644
--- a/net/ipv4/tcp_input.c
+++ b/net/ipv4/tcp_input.c
@@ -4535,26 +4535,13 @@ restart:
return;
while (before(start, end)) {
+ int copy = min_t(int, SKB_MAX_ORDER(0, 0), end - start);
struct sk_buff *nskb;
- unsigned int header = skb_headroom(skb);
- int copy = SKB_MAX_ORDER(header, 0);
- /* Too big header? This can happen with IPv6. */
- if (copy < 0)
- return;
- if (end - start < copy)
- copy = end - start;
- nskb = alloc_skb(copy + header, GFP_ATOMIC);
+ nskb = alloc_skb(copy, GFP_ATOMIC);
if (!nskb)
return;
- skb_set_mac_header(nskb, skb_mac_header(skb) - skb->head);
- skb_set_network_header(nskb, (skb_network_header(skb) -
- skb->head));
- skb_set_transport_header(nskb, (skb_transport_header(skb) -
- skb->head));
- skb_reserve(nskb, header);
- memcpy(nskb->head, skb->head, header);
memcpy(nskb->cb, skb->cb, sizeof(skb->cb));
TCP_SKB_CB(nskb)->seq = TCP_SKB_CB(nskb)->end_seq = start;
__skb_queue_before(list, skb, nskb);
--
2.1.0.rc2.206.gedb03e5
^ permalink raw reply related
* [PATCH v2 net-next 2/3] tcp: allow segment with FIN in tcp_try_coalesce()
From: Eric Dumazet @ 2014-09-15 11:19 UTC (permalink / raw)
To: David S. Miller; +Cc: netdev, Yuchung Cheng, Neal Cardwell, Eric Dumazet
In-Reply-To: <1410779993-31464-1-git-send-email-edumazet@google.com>
We can allow a segment with FIN to be aggregated,
if we take care to add tcp flags,
and if skb_try_coalesce() takes care of zero sized skbs.
Signed-off-by: Eric Dumazet <edumazet@google.com>
---
net/core/skbuff.c | 3 ++-
net/ipv4/tcp_input.c | 4 +---
2 files changed, 3 insertions(+), 4 deletions(-)
diff --git a/net/core/skbuff.c b/net/core/skbuff.c
index c8259ac38745..29f7f0121491 100644
--- a/net/core/skbuff.c
+++ b/net/core/skbuff.c
@@ -3936,7 +3936,8 @@ bool skb_try_coalesce(struct sk_buff *to, struct sk_buff *from,
return false;
if (len <= skb_tailroom(to)) {
- BUG_ON(skb_copy_bits(from, 0, skb_put(to, len), len));
+ if (len)
+ BUG_ON(skb_copy_bits(from, 0, skb_put(to, len), len));
*delta_truesize = 0;
return true;
}
diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c
index 8f639a4face9..228bf0c5ff19 100644
--- a/net/ipv4/tcp_input.c
+++ b/net/ipv4/tcp_input.c
@@ -4143,9 +4143,6 @@ static bool tcp_try_coalesce(struct sock *sk,
*fragstolen = false;
- if (tcp_hdr(from)->fin)
- return false;
-
/* Its possible this segment overlaps with prior segment in queue */
if (TCP_SKB_CB(from)->seq != TCP_SKB_CB(to)->end_seq)
return false;
@@ -4158,6 +4155,7 @@ static bool tcp_try_coalesce(struct sock *sk,
NET_INC_STATS_BH(sock_net(sk), LINUX_MIB_TCPRCVCOALESCE);
TCP_SKB_CB(to)->end_seq = TCP_SKB_CB(from)->end_seq;
TCP_SKB_CB(to)->ack_seq = TCP_SKB_CB(from)->ack_seq;
+ TCP_SKB_CB(to)->tcp_flags |= TCP_SKB_CB(from)->tcp_flags;
return true;
}
--
2.1.0.rc2.206.gedb03e5
^ permalink raw reply related
* [PATCH v2 net-next 1/3] tcp: use TCP_SKB_CB(skb)->tcp_flags in input path
From: Eric Dumazet @ 2014-09-15 11:19 UTC (permalink / raw)
To: David S. Miller; +Cc: netdev, Yuchung Cheng, Neal Cardwell, Eric Dumazet
In-Reply-To: <1410779993-31464-1-git-send-email-edumazet@google.com>
Input path of TCP do not currently uses TCP_SKB_CB(skb)->tcp_flags,
which is only used in output path.
tcp_recvmsg(), looks at tcp_hdr(skb)->syn for every skb found in receive queue,
and its unfortunate because this bit is located in a cache line right before
the payload.
We can simplify TCP by copying tcp flags into TCP_SKB_CB(skb)->tcp_flags.
This patch does so, and avoids the cache line miss in tcp_recvmsg()
Following patches will
- allow a segment with FIN being coalesced in tcp_try_coalesce()
- simplify tcp_collapse() by not copying the headers.
Signed-off-by: Eric Dumazet <edumazet@google.com>
---
net/ipv4/tcp.c | 18 ++++++++++--------
net/ipv4/tcp_input.c | 10 +++++-----
net/ipv4/tcp_ipv4.c | 1 +
net/ipv6/tcp_ipv6.c | 1 +
4 files changed, 17 insertions(+), 13 deletions(-)
diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c
index 541f26a67ba2..070aeff1b131 100644
--- a/net/ipv4/tcp.c
+++ b/net/ipv4/tcp.c
@@ -1510,9 +1510,9 @@ static struct sk_buff *tcp_recv_skb(struct sock *sk, u32 seq, u32 *off)
while ((skb = skb_peek(&sk->sk_receive_queue)) != NULL) {
offset = seq - TCP_SKB_CB(skb)->seq;
- if (tcp_hdr(skb)->syn)
+ if (TCP_SKB_CB(skb)->tcp_flags & TCPHDR_SYN)
offset--;
- if (offset < skb->len || tcp_hdr(skb)->fin) {
+ if (offset < skb->len || (TCP_SKB_CB(skb)->tcp_flags & TCPHDR_FIN)) {
*off = offset;
return skb;
}
@@ -1585,7 +1585,7 @@ int tcp_read_sock(struct sock *sk, read_descriptor_t *desc,
if (offset + 1 != skb->len)
continue;
}
- if (tcp_hdr(skb)->fin) {
+ if (TCP_SKB_CB(skb)->tcp_flags & TCPHDR_FIN) {
sk_eat_skb(sk, skb, false);
++seq;
break;
@@ -1722,11 +1722,11 @@ int tcp_recvmsg(struct kiocb *iocb, struct sock *sk, struct msghdr *msg,
break;
offset = *seq - TCP_SKB_CB(skb)->seq;
- if (tcp_hdr(skb)->syn)
+ if (TCP_SKB_CB(skb)->tcp_flags & TCPHDR_SYN)
offset--;
if (offset < skb->len)
goto found_ok_skb;
- if (tcp_hdr(skb)->fin)
+ if (TCP_SKB_CB(skb)->tcp_flags & TCPHDR_FIN)
goto found_fin_ok;
WARN(!(flags & MSG_PEEK),
"recvmsg bug 2: copied %X seq %X rcvnxt %X fl %X\n",
@@ -1959,7 +1959,7 @@ skip_copy:
if (used + offset < skb->len)
continue;
- if (tcp_hdr(skb)->fin)
+ if (TCP_SKB_CB(skb)->tcp_flags & TCPHDR_FIN)
goto found_fin_ok;
if (!(flags & MSG_PEEK)) {
sk_eat_skb(sk, skb, copied_early);
@@ -2160,8 +2160,10 @@ void tcp_close(struct sock *sk, long timeout)
* reader process may not have drained the data yet!
*/
while ((skb = __skb_dequeue(&sk->sk_receive_queue)) != NULL) {
- u32 len = TCP_SKB_CB(skb)->end_seq - TCP_SKB_CB(skb)->seq -
- tcp_hdr(skb)->fin;
+ u32 len = TCP_SKB_CB(skb)->end_seq - TCP_SKB_CB(skb)->seq;
+
+ if (TCP_SKB_CB(skb)->tcp_flags & TCPHDR_FIN)
+ len--;
data_was_unread += len;
__kfree_skb(skb);
}
diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c
index f97003ad0af5..8f639a4face9 100644
--- a/net/ipv4/tcp_input.c
+++ b/net/ipv4/tcp_input.c
@@ -4093,7 +4093,7 @@ static void tcp_ofo_queue(struct sock *sk)
__skb_unlink(skb, &tp->out_of_order_queue);
__skb_queue_tail(&sk->sk_receive_queue, skb);
tp->rcv_nxt = TCP_SKB_CB(skb)->end_seq;
- if (tcp_hdr(skb)->fin)
+ if (TCP_SKB_CB(skb)->tcp_flags & TCPHDR_FIN)
tcp_fin(sk);
}
}
@@ -4513,7 +4513,7 @@ restart:
* - bloated or contains data before "start" or
* overlaps to the next one.
*/
- if (!tcp_hdr(skb)->syn && !tcp_hdr(skb)->fin &&
+ if (!(TCP_SKB_CB(skb)->tcp_flags & (TCPHDR_SYN | TCPHDR_FIN)) &&
(tcp_win_from_space(skb->truesize) > skb->len ||
before(TCP_SKB_CB(skb)->seq, start))) {
end_of_skbs = false;
@@ -4532,7 +4532,8 @@ restart:
/* Decided to skip this, advance start seq. */
start = TCP_SKB_CB(skb)->end_seq;
}
- if (end_of_skbs || tcp_hdr(skb)->syn || tcp_hdr(skb)->fin)
+ if (end_of_skbs ||
+ (TCP_SKB_CB(skb)->tcp_flags & (TCPHDR_SYN | TCPHDR_FIN)))
return;
while (before(start, end)) {
@@ -4579,8 +4580,7 @@ restart:
skb = tcp_collapse_one(sk, skb, list);
if (!skb ||
skb == tail ||
- tcp_hdr(skb)->syn ||
- tcp_hdr(skb)->fin)
+ (TCP_SKB_CB(skb)->tcp_flags & (TCPHDR_SYN | TCPHDR_FIN)))
return;
}
}
diff --git a/net/ipv4/tcp_ipv4.c b/net/ipv4/tcp_ipv4.c
index 7881b96d2b72..006b045716d8 100644
--- a/net/ipv4/tcp_ipv4.c
+++ b/net/ipv4/tcp_ipv4.c
@@ -1638,6 +1638,7 @@ int tcp_v4_rcv(struct sk_buff *skb)
TCP_SKB_CB(skb)->end_seq = (TCP_SKB_CB(skb)->seq + th->syn + th->fin +
skb->len - th->doff * 4);
TCP_SKB_CB(skb)->ack_seq = ntohl(th->ack_seq);
+ TCP_SKB_CB(skb)->tcp_flags = tcp_flag_byte(th);
TCP_SKB_CB(skb)->tcp_tw_isn = 0;
TCP_SKB_CB(skb)->ip_dsfield = ipv4_get_dsfield(iph);
TCP_SKB_CB(skb)->sacked = 0;
diff --git a/net/ipv6/tcp_ipv6.c b/net/ipv6/tcp_ipv6.c
index 1835480336ac..de51a88bec6f 100644
--- a/net/ipv6/tcp_ipv6.c
+++ b/net/ipv6/tcp_ipv6.c
@@ -1415,6 +1415,7 @@ static int tcp_v6_rcv(struct sk_buff *skb)
TCP_SKB_CB(skb)->end_seq = (TCP_SKB_CB(skb)->seq + th->syn + th->fin +
skb->len - th->doff*4);
TCP_SKB_CB(skb)->ack_seq = ntohl(th->ack_seq);
+ TCP_SKB_CB(skb)->tcp_flags = tcp_flag_byte(th);
TCP_SKB_CB(skb)->tcp_tw_isn = 0;
TCP_SKB_CB(skb)->ip_dsfield = ipv6_get_dsfield(hdr);
TCP_SKB_CB(skb)->sacked = 0;
--
2.1.0.rc2.206.gedb03e5
^ permalink raw reply related
* [PATCH v2 net-next 0/3] tcp: no longer keep around headers in input path
From: Eric Dumazet @ 2014-09-15 11:19 UTC (permalink / raw)
To: David S. Miller; +Cc: netdev, Yuchung Cheng, Neal Cardwell, Eric Dumazet
Looking at tcp_try_coalesce() I was wondering why I did :
if (tcp_hdr(from)->fin)
return false;
The answer would be to allow the aggregation, if we simply OR the FIN and PSH
flags eventually present in @from to @to packet. (Note a change is also
needed in skb_try_coalesce() to avoid calling skb_put() with 0 len)
Then, looking at tcp_recvmsg(), I realized we access tcp_hdr(skb)->syn
(and maybe tcp_hdr(skb)->fin) for every packet we process from socket
receive queue.
We have to understand TCP flags are cold in cpu caches most of the time
(assuming TCP timestamps, and that application calls recvmsg() a long
time after incoming packet was processed), and bringing a whole
cache line only to access one bit is not very nice.
It would make sense to use in TCP input path TCP_SKB_CB(skb)->tcp_flags
as we do in output path.
This saves one cache line miss, and TCP tcp_collapse() can avoid dealing
with the headers.
Eric Dumazet (3):
tcp: use TCP_SKB_CB(skb)->tcp_flags in input path
tcp: allow segment with FIN in tcp_try_coalesce()
tcp: do not copy headers in tcp_collapse()
net/core/skbuff.c | 3 ++-
net/ipv4/tcp.c | 18 ++++++++++--------
net/ipv4/tcp_input.c | 31 ++++++++-----------------------
net/ipv4/tcp_ipv4.c | 1 +
net/ipv6/tcp_ipv6.c | 1 +
5 files changed, 22 insertions(+), 32 deletions(-)
--
2.1.0.rc2.206.gedb03e5
^ permalink raw reply
* Re: [PATCH] iproute: update dsfield file values
From: Jesper Dangaard Brouer @ 2014-09-15 11:06 UTC (permalink / raw)
To: Stephen Hemminger; +Cc: brouer, netdev
In-Reply-To: <20140914204353.5a0579be@urahara>
On Sun, 14 Sep 2014 20:43:53 -0700
Stephen Hemminger <stephen@networkplumber.org> wrote:
> Update the rt_dsfield file to contain values defined in current RFC.
> The days of TOS precedence are gone, even Cisco doesn't refer
> to these in the documents.
> ---
> etc/iproute2/rt_dsfield | 26 ++++++++++++--------------
> 1 file changed, 12 insertions(+), 14 deletions(-)
>
> diff --git a/etc/iproute2/rt_dsfield b/etc/iproute2/rt_dsfield
> index 496ef66..c0f3679 100644
> --- a/etc/iproute2/rt_dsfield
> +++ b/etc/iproute2/rt_dsfield
> @@ -1,17 +1,6 @@
> -0x00 default
> -0x10 lowdelay
SSH clients still set ToS bits to 0x10.
And our default pfifo_fast qdisc puts this into the high-prio band.
(TC_PRIO_INTERACTIVE=6 mapped via prio2band[16] to band 0)
rt_tos2priority() lookup bit masks "tos & 0x1E"
(and perform mapping lookup (note shift right(tos)>>1) in ip_tos2prio[16])
Maybe the kernel is handling DSCP wrongly? see below.
> -0x08 throughput
> -0x04 reliability
> -# This value overlap with ECT, do not use it!
> -0x02 mincost
> -# These values seems do not want to die, Cisco likes them by a strange reason.
> -0x20 priority
> -0x40 immediate
> -0x60 flash
> -0x80 flash-override
> -0xa0 critical
> -0xc0 internet
> -0xe0 network
> +# Differentiated field values
> +# These include the DSCP and unused bits
> +0x0 default
> # Newer RFC2597 values
> 0x28 AF11
> 0x30 AF12
E.g. kernel will map AF12 to TC_PRIO_INTERACTIVE
((0x30 & 0x1E)>>1) => 8
lookup in ip_tos2prio[8] = TC_PRIO_INTERACTIVE
> @@ -25,3 +14,12 @@
> 0x88 AF41
> 0x90 AF42
> 0x98 AF43
> +# Older values RFC2474
> +0x20 CS1
> +0x40 CS2
> +0x60 CS3
> +0x80 CS4
> +0xA0 CS5
> +0xC0 CS6
> +0xE0 CS7
> +0x5C EF
(bash)$ echo $(((0x5C & 0x1E)>>1)) # = 14
ip_tos2prio[14] = TC_PRIO_INTERACTIVE_BULK
Is it a bug, that we put DSCP's Expedited Forwarding (EF) into the
best-effort priority band(1) in pfifo_fast???
Below diff that highlight the kernel code doing the mapping:
[PATCH] net: tracking to ToS/DSCP values are mapped to pfifo_fast prio bands
From: Jesper Dangaard Brouer <brouer@redhat.com>
Something seems wrong with the mapping of DSCP mappings to prio bands
in our default pfifo_fast qdisc.
Especially it seems strange, that DSCP's Expedited Forwarding (EF)
get mapped into the best-effort priority band(1) in pfifo_fast.
---
include/net/route.h | 2 +-
include/uapi/linux/ip.h | 3 ++-
include/uapi/linux/pkt_sched.h | 1 +
net/ipv4/route.c | 35 ++++++++++++++++++-----------------
net/sched/sch_generic.c | 7 ++++++-
5 files changed, 28 insertions(+), 20 deletions(-)
diff --git a/include/net/route.h b/include/net/route.h
index b17cf28..6fc63e2 100644
--- a/include/net/route.h
+++ b/include/net/route.h
@@ -208,7 +208,7 @@ static inline void ip_rt_put(struct rtable *rt)
#define IPTOS_RT_MASK (IPTOS_TOS_MASK & ~3)
extern const __u8 ip_tos2prio[16];
-
+// hey diff
static inline char rt_tos2priority(u8 tos)
{
return ip_tos2prio[IPTOS_TOS(tos)>>1];
diff --git a/include/uapi/linux/ip.h b/include/uapi/linux/ip.h
index 4119594..36849ff 100644
--- a/include/uapi/linux/ip.h
+++ b/include/uapi/linux/ip.h
@@ -18,10 +18,11 @@
#define _UAPI_LINUX_IP_H
#include <linux/types.h>
#include <asm/byteorder.h>
-
+// hey diff here is the IPTOS_TOS_MASK
#define IPTOS_TOS_MASK 0x1E
#define IPTOS_TOS(tos) ((tos)&IPTOS_TOS_MASK)
#define IPTOS_LOWDELAY 0x10
+// Hmmm, is IPTOS_LOWDELAY exported to userspace?
#define IPTOS_THROUGHPUT 0x08
#define IPTOS_RELIABILITY 0x04
#define IPTOS_MINCOST 0x02
diff --git a/include/uapi/linux/pkt_sched.h b/include/uapi/linux/pkt_sched.h
index d62316b..d07322b 100644
--- a/include/uapi/linux/pkt_sched.h
+++ b/include/uapi/linux/pkt_sched.h
@@ -19,6 +19,7 @@
#define TC_PRIO_BESTEFFORT 0
#define TC_PRIO_FILLER 1
#define TC_PRIO_BULK 2
+// hey diff look at priority defines here
#define TC_PRIO_INTERACTIVE_BULK 4
#define TC_PRIO_INTERACTIVE 6
#define TC_PRIO_CONTROL 7
diff --git a/net/ipv4/route.c b/net/ipv4/route.c
index eaa4b00..0d547c1 100644
--- a/net/ipv4/route.c
+++ b/net/ipv4/route.c
@@ -168,23 +168,24 @@ static struct dst_ops ipv4_dst_ops = {
#define ECN_OR_COST(class) TC_PRIO_##class
-const __u8 ip_tos2prio[16] = {
- TC_PRIO_BESTEFFORT,
- ECN_OR_COST(BESTEFFORT),
- TC_PRIO_BESTEFFORT,
- ECN_OR_COST(BESTEFFORT),
- TC_PRIO_BULK,
- ECN_OR_COST(BULK),
- TC_PRIO_BULK,
- ECN_OR_COST(BULK),
- TC_PRIO_INTERACTIVE,
- ECN_OR_COST(INTERACTIVE),
- TC_PRIO_INTERACTIVE,
- ECN_OR_COST(INTERACTIVE),
- TC_PRIO_INTERACTIVE_BULK,
- ECN_OR_COST(INTERACTIVE_BULK),
- TC_PRIO_INTERACTIVE_BULK,
- ECN_OR_COST(INTERACTIVE_BULK)
+/* lookup: tos bitmasked 0x1E and shifted right (tos>>1) in rt_tos2priority(tos) */
+const __u8 ip_tos2prio[16] = { // lower-two bits should have been for ECN (see "-" split)
+ TC_PRIO_BESTEFFORT, // [0] 000-00 = 0x00 default
+ ECN_OR_COST(BESTEFFORT), // [1] 000-10 = 0x02 TOS-"mincost" (conflict with ECN bits)
+ TC_PRIO_BESTEFFORT, // [2] 001-00 = 0x04 TOS-"reliability"
+ ECN_OR_COST(BESTEFFORT), // [3] 001-10 = 0x06
+ TC_PRIO_BULK, // [4] 010-00 = 0x08 TOS-"throughput" DSCP(AF11+21+31+41)
+ ECN_OR_COST(BULK), // [5] 010-10 = 0x0A
+ TC_PRIO_BULK, // [6] 011-00 = 0x0C
+ ECN_OR_COST(BULK), // [7] 011-10 = 0X0E
+ TC_PRIO_INTERACTIVE, // [8] 100-00 = 0x10 TOS-"lowdelay" - DSCP(AF12+22+32+42)
+ ECN_OR_COST(INTERACTIVE), // [9] 100-10 = 0x12
+ TC_PRIO_INTERACTIVE, // [10] 101-00 = 0x14
+ ECN_OR_COST(INTERACTIVE), // [11] 101-10 = 0x16
+ TC_PRIO_INTERACTIVE_BULK, // [12] 110-00 = 0x18 DSCP(AF13+23+33+43)
+ ECN_OR_COST(INTERACTIVE_BULK),// [13] 110-10 = 0x1A
+ TC_PRIO_INTERACTIVE_BULK, // [14] 111-00 = 0x1C DSCP(EF) (0x5C & 0x1E)=0x1C
+ ECN_OR_COST(INTERACTIVE_BULK) // [15] 111-10 = 0x1E
};
EXPORT_SYMBOL(ip_tos2prio);
diff --git a/net/sched/sch_generic.c b/net/sched/sch_generic.c
index a8bf9f9..c353234 100644
--- a/net/sched/sch_generic.c
+++ b/net/sched/sch_generic.c
@@ -408,7 +408,12 @@ static struct Qdisc noqueue_qdisc = {
.busylock = __SPIN_LOCK_UNLOCKED(noqueue_qdisc.busylock),
};
-
+// ToS 0x10 maps to TC_PRIO_INTERACTIVE=6 => band 0
+// DSCP(EF) maps to TC_PRIO_INTERACTIVE_BULK=4 => band 1
+//
+// DSCP(AF11+21+31+41) TC_PRIO_BULK=2 => band 2
+// DSCP(AF12+22+32+42) TC_PRIO_INTERACTIVE=6 => band 0
+// DSCP(AF13+23+33+43) TC_PRIO_INTERACTIVE_BULK=4 => band 1
static const u8 prio2band[TC_PRIO_MAX + 1] = {
1, 2, 2, 2, 1, 2, 0, 0 , 1, 1, 1, 1, 1, 1, 1, 1
};
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Sr. Network Kernel Developer at Red Hat
Author of http://www.iptv-analyzer.org
LinkedIn: http://www.linkedin.com/in/brouer
^ permalink raw reply related
* Re: [PATCH net-next 2/3] tcp: allow segment with FIN in tcp_try_coalesce()
From: Eric Dumazet @ 2014-09-15 10:57 UTC (permalink / raw)
To: Eric Dumazet; +Cc: David S. Miller, netdev, Yuchung Cheng, Neal Cardwell
In-Reply-To: <1410749780-30613-3-git-send-email-edumazet@google.com>
On Sun, 2014-09-14 at 19:56 -0700, Eric Dumazet wrote:
> There is nothing special with allowing a segment with FIN to be aggregated,
> if we take care to add tcp flags.
>
> Signed-of-by: Eric Dumazet <edumazet@google.com>
> ---
> net/ipv4/tcp_input.c | 4 +---
> 1 file changed, 1 insertion(+), 3 deletions(-)
>
> diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c
> index 8f639a4face9..228bf0c5ff19 100644
> --- a/net/ipv4/tcp_input.c
> +++ b/net/ipv4/tcp_input.c
> @@ -4143,9 +4143,6 @@ static bool tcp_try_coalesce(struct sock *sk,
>
> *fragstolen = false;
>
> - if (tcp_hdr(from)->fin)
> - return false;
> -
> /* Its possible this segment overlaps with prior segment in queue */
> if (TCP_SKB_CB(from)->seq != TCP_SKB_CB(to)->end_seq)
> return false;
> @@ -4158,6 +4155,7 @@ static bool tcp_try_coalesce(struct sock *sk,
> NET_INC_STATS_BH(sock_net(sk), LINUX_MIB_TCPRCVCOALESCE);
> TCP_SKB_CB(to)->end_seq = TCP_SKB_CB(from)->end_seq;
> TCP_SKB_CB(to)->ack_seq = TCP_SKB_CB(from)->ack_seq;
> + TCP_SKB_CB(to)->tcp_flags |= TCP_SKB_CB(from)->tcp_flags;
> return true;
> }
>
This patch needs to be refined.
skb_try_coalesce() should not do an unconditional
BUG_ON(skb_copy_bits(from, 0, skb_put(to, len), len));
And do it only if len is non 0
I will send a v2
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox