* Re: [PATCH v4 net-next 0/5] bonding:use latest lacp_rate and ad_select and delete unused ad_timer and arp_mon_pt
From: David Miller @ 2011-06-09 7:21 UTC (permalink / raw)
To: panweiping3; +Cc: fubar, andy, netdev, linux-kernel
In-Reply-To: <cover.1307603621.git.panweiping3@gmail.com>
My head will spin if I see yet another submission of these
patches :-(
^ permalink raw reply
* Re: [PATCH net-next 0/2] bonding: delete two unused variables
From: WeipingPan @ 2011-06-09 7:23 UTC (permalink / raw)
To: Américo Wang; +Cc: fubar, andy, netdev, linux-kernel
In-Reply-To: <BANLkTi=xoCJUZAfHBR18bs_WmXwD7rt58g@mail.gmail.com>
On 06/09/2011 11:19 AM, Américo Wang wrote:
> On Thu, Jun 9, 2011 at 10:51 AM, Weiping Pan<panweiping3@gmail.com> wrote:
>> Delete two unused variables in bonding.
>>
>> Weiping Pan (2):
>> bonding: delete unused ad_timer
>> bonding: delete unused arp_mon_pt
>>
> Both look good to me,
>
> Reviewed-by: WANG Cong<xiyou.wangcong@gmail.com>
>
> Thanks.
thanks,
But I move these two patches into another patchset,
the title is "[PATCH v4 net-next 0/5] bonding:use latest lacp_rate and
ad_select and delete unused ad_timer and arp_mon_pt"
Hope it will be convenient to be reviewed and merged, so forget these
two patches.
thanks
Weiping Pan
^ permalink raw reply
* Re: [PATCH net-next-2.6] net: pmtu_expires fixes
From: David Miller @ 2011-06-09 7:25 UTC (permalink / raw)
To: eric.dumazet; +Cc: netdev
In-Reply-To: <1307549227.3057.64.camel@edumazet-laptop>
From: Eric Dumazet <eric.dumazet@gmail.com>
Date: Wed, 08 Jun 2011 18:07:07 +0200
> commit 2c8cec5c10bc (ipv4: Cache learned PMTU information in inetpeer)
> added some racy peer->pmtu_expires accesses.
>
> As its value can be changed by another cpu/thread, we should be more
> careful, reading its value once.
>
> Add peer_pmtu_expired() and peer_pmtu_cleaned() helpers
>
> Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
I've decided to apply this to net-2.6, thanks Eric.
^ permalink raw reply
* Re: [PATCH] tun: do not put self in waitq if doing a nonblock read
From: David Miller @ 2011-06-09 7:27 UTC (permalink / raw)
To: akong; +Cc: netdev, jasowang, kvm, mst
In-Reply-To: <20110608234606.8681.19932.stgit@localhost6.localdomain6>
From: Amos Kong <akong@redhat.com>
Date: Thu, 09 Jun 2011 07:46:06 +0800
> Perf shows a relatively high rate (about 8%) race in
> spin_lock_irqsave() when doing netperf between external host and
> guest. It's mainly becuase the lock contention between the
> tun_do_read() and tun_xmit_skb(), so this patch do not put self into
> waitqueue to reduce this kind of race. After this patch, it drops to
> 4%.
>
> Signed-off-by: Jason Wang <jasowang@redhat.com>
> Signed-off-by: Amos Kong <akong@redhat.com>
Applied, thanks.
^ permalink raw reply
* Re: [PATCH v4 net-next 0/5] bonding:use latest lacp_rate and ad_select and delete unused ad_timer and arp_mon_pt
From: WeipingPan @ 2011-06-09 7:29 UTC (permalink / raw)
To: David Miller; +Cc: fubar, andy, netdev, linux-kernel
In-Reply-To: <20110609.002155.1433667911487783052.davem@davemloft.net>
On 06/09/2011 03:21 PM, David Miller wrote:
> My head will spin if I see yet another submission of these
> patches :-(
sorry for the inconvenience
Weiping Pan
^ permalink raw reply
* Re: [Patch] netpoll: prevent netpoll setup on slave devices
From: David Miller @ 2011-06-09 7:28 UTC (permalink / raw)
To: amwang; +Cc: netdev, nhorman
In-Reply-To: <1307587360-3092-1-git-send-email-amwang@redhat.com>
From: Amerigo Wang <amwang@redhat.com>
Date: Thu, 9 Jun 2011 10:42:40 +0800
> In commit 8d8fc29d02a33e4bd5f4fa47823c1fd386346093
> (netpoll: disable netpoll when enslave a device), we automatically
> disable netpoll when the underlying device is being enslaved,
> we also need to prevent people from setuping netpoll on
> devices that are already enslaved.
>
> Signed-off-by: WANG Cong <amwang@redhat.com>
Applied, thanks.
^ permalink raw reply
* RE: [PATCH] gianfar:localized filer table
From: Sebastian Pöhn @ 2011-06-09 7:30 UTC (permalink / raw)
To: b06378; +Cc: netdev, sebastian.poehn
[-- Attachment #1: Type: text/plain, Size: 11446 bytes --]
Looks good so far. Maybe you want by the way fix the compiler warning
caused by the huge local_rqfpr and local_rqfcr arrays?
drivers/net/gianfar_ethtool.c: In function 'gfar_ethflow_to_filer_table':
drivers/net/gianfar_ethtool.c:764:1: warning: the frame size of 2048 bytes
is larger than 1024 bytes
I am currently implementing rx_ntuple for gianfar (now i have to change it
to rxcls). At the moment I use the whole rx queue filer table for my
purposes and overwrite the whole rxfh stuff. So you only may use either
RXFH or RXCLS because dividing the table to be used by both simultaneously
would be quite challenging.
-----netdev-owner@vger.kernel.org schrieb: -----
An: <netdev@vger.kernel.org>, <davem@davemloft.net>
Von: Jiajun Wu
Gesendet von: netdev-owner@vger.kernel.org
Datum: 08.06.2011 10:39
Kopie: <linuxppc-dev@lists.ozlabs.org>, Jiajun Wu <b06378@freescale.com>
Betreff: [PATCH] gianfar:localized filer table
Each eTSEC device should own localized filer table.
Signed-off-by: Jiajun Wu <b06378@freescale.com>
---
drivers/net/gianfar.c | 29 ++++++++----------
drivers/net/gianfar.h | 8 +++--
drivers/net/gianfar_ethtool.c | 64
+++++++++++++++++++++--------------------
3 files changed, 51 insertions(+), 50 deletions(-)
diff --git a/drivers/net/gianfar.c b/drivers/net/gianfar.c
index ff60b23..2dfcc80 100644
--- a/drivers/net/gianfar.c
+++ b/drivers/net/gianfar.c
@@ -10,7 +10,7 @@
* Maintainer: Kumar Gala
* Modifier: Sandeep Gopalpet <sandeep.kumar@freescale.com>
*
- * Copyright 2002-2009 Freescale Semiconductor, Inc.
+ * Copyright 2002-2009, 2011 Freescale Semiconductor, Inc.
* Copyright 2007 MontaVista Software, Inc.
*
* This program is free software; you can redistribute it and/or modify it
@@ -476,9 +476,6 @@ static const struct net_device_ops gfar_netdev_ops = {
#endif
};
-unsigned int ftp_rqfpr[MAX_FILER_IDX + 1];
-unsigned int ftp_rqfcr[MAX_FILER_IDX + 1];
-
void lock_rx_qs(struct gfar_private *priv)
{
int i = 0x0;
@@ -868,28 +865,28 @@ static u32 cluster_entry_per_class(struct
gfar_private *priv, u32 rqfar,
rqfar--;
rqfcr = RQFCR_CLE | RQFCR_PID_MASK | RQFCR_CMP_EXACT;
- ftp_rqfpr[rqfar] = rqfpr;
- ftp_rqfcr[rqfar] = rqfcr;
+ priv->ftp_rqfpr[rqfar] = rqfpr;
+ priv->ftp_rqfcr[rqfar] = rqfcr;
gfar_write_filer(priv, rqfar, rqfcr, rqfpr);
rqfar--;
rqfcr = RQFCR_CMP_NOMATCH;
- ftp_rqfpr[rqfar] = rqfpr;
- ftp_rqfcr[rqfar] = rqfcr;
+ priv->ftp_rqfpr[rqfar] = rqfpr;
+ priv->ftp_rqfcr[rqfar] = rqfcr;
gfar_write_filer(priv, rqfar, rqfcr, rqfpr);
rqfar--;
rqfcr = RQFCR_CMP_EXACT | RQFCR_PID_PARSE | RQFCR_CLE | RQFCR_AND;
rqfpr = class;
- ftp_rqfcr[rqfar] = rqfcr;
- ftp_rqfpr[rqfar] = rqfpr;
+ priv->ftp_rqfcr[rqfar] = rqfcr;
+ priv->ftp_rqfpr[rqfar] = rqfpr;
gfar_write_filer(priv, rqfar, rqfcr, rqfpr);
rqfar--;
rqfcr = RQFCR_CMP_EXACT | RQFCR_PID_MASK | RQFCR_AND;
rqfpr = class;
- ftp_rqfcr[rqfar] = rqfcr;
- ftp_rqfpr[rqfar] = rqfpr;
+ priv->ftp_rqfcr[rqfar] = rqfcr;
+ priv->ftp_rqfpr[rqfar] = rqfpr;
gfar_write_filer(priv, rqfar, rqfcr, rqfpr);
return rqfar;
@@ -904,8 +901,8 @@ static void gfar_init_filer_table(struct gfar_private
*priv)
/* Default rule */
rqfcr = RQFCR_CMP_MATCH;
- ftp_rqfcr[rqfar] = rqfcr;
- ftp_rqfpr[rqfar] = rqfpr;
+ priv->ftp_rqfcr[rqfar] = rqfcr;
+ priv->ftp_rqfpr[rqfar] = rqfpr;
gfar_write_filer(priv, rqfar, rqfcr, rqfpr);
rqfar = cluster_entry_per_class(priv, rqfar, RQFPR_IPV6);
@@ -921,8 +918,8 @@ static void gfar_init_filer_table(struct gfar_private
*priv)
/* Rest are masked rules */
rqfcr = RQFCR_CMP_NOMATCH;
for (i = 0; i < rqfar; i++) {
- ftp_rqfcr[i] = rqfcr;
- ftp_rqfpr[i] = rqfpr;
+ priv->ftp_rqfcr[i] = rqfcr;
+ priv->ftp_rqfpr[i] = rqfpr;
gfar_write_filer(priv, i, rqfcr, rqfpr);
}
}
diff --git a/drivers/net/gianfar.h b/drivers/net/gianfar.h
index fc86f51..ba36dc7 100644
--- a/drivers/net/gianfar.h
+++ b/drivers/net/gianfar.h
@@ -9,7 +9,7 @@
* Maintainer: Kumar Gala
* Modifier: Sandeep Gopalpet <sandeep.kumar@freescale.com>
*
- * Copyright 2002-2009 Freescale Semiconductor, Inc.
+ * Copyright 2002-2009, 2011 Freescale Semiconductor, Inc.
*
* This program is free software; you can redistribute it and/or modify it
* under the terms of the GNU General Public License as published by the
@@ -1107,10 +1107,12 @@ struct gfar_private {
/* HW time stamping enabled flag */
int hwts_rx_en;
int hwts_tx_en;
+
+ /*Filer table*/
+ unsigned int ftp_rqfpr[MAX_FILER_IDX + 1];
+ unsigned int ftp_rqfcr[MAX_FILER_IDX + 1];
};
-extern unsigned int ftp_rqfpr[MAX_FILER_IDX + 1];
-extern unsigned int ftp_rqfcr[MAX_FILER_IDX + 1];
static inline int gfar_has_errata(struct gfar_private *priv,
enum gfar_errata err)
diff --git a/drivers/net/gianfar_ethtool.c b/drivers/net/gianfar_ethtool.c
index 493d743..239e333 100644
--- a/drivers/net/gianfar_ethtool.c
+++ b/drivers/net/gianfar_ethtool.c
@@ -9,7 +9,7 @@
* Maintainer: Kumar Gala
* Modifier: Sandeep Gopalpet <sandeep.kumar@freescale.com>
*
- * Copyright 2003-2006, 2008-2009 Freescale Semiconductor, Inc.
+ * Copyright 2003-2006, 2008-2009, 2011 Freescale Semiconductor, Inc.
*
* This software may be used and distributed according to
* the terms of the GNU Public License, Version 2, incorporated herein
@@ -609,15 +609,15 @@ static void ethflow_to_filer_rules (struct
gfar_private *priv, u64 ethflow)
if (ethflow & RXH_L2DA) {
fcr = RQFCR_PID_DAH |RQFCR_CMP_NOMATCH |
RQFCR_HASH | RQFCR_AND | RQFCR_HASHTBL_0;
- ftp_rqfpr[priv->cur_filer_idx] = fpr;
- ftp_rqfcr[priv->cur_filer_idx] = fcr;
+ priv->ftp_rqfpr[priv->cur_filer_idx] = fpr;
+ priv->ftp_rqfcr[priv->cur_filer_idx] = fcr;
gfar_write_filer(priv, priv->cur_filer_idx, fcr, fpr);
priv->cur_filer_idx = priv->cur_filer_idx - 1;
fcr = RQFCR_PID_DAL | RQFCR_AND | RQFCR_CMP_NOMATCH |
RQFCR_HASH | RQFCR_AND | RQFCR_HASHTBL_0;
- ftp_rqfpr[priv->cur_filer_idx] = fpr;
- ftp_rqfcr[priv->cur_filer_idx] = fcr;
+ priv->ftp_rqfpr[priv->cur_filer_idx] = fpr;
+ priv->ftp_rqfcr[priv->cur_filer_idx] = fcr;
gfar_write_filer(priv, priv->cur_filer_idx, fcr, fpr);
priv->cur_filer_idx = priv->cur_filer_idx - 1;
}
@@ -626,16 +626,16 @@ static void ethflow_to_filer_rules (struct
gfar_private *priv, u64 ethflow)
fcr = RQFCR_PID_VID | RQFCR_CMP_NOMATCH | RQFCR_HASH |
RQFCR_AND | RQFCR_HASHTBL_0;
gfar_write_filer(priv, priv->cur_filer_idx, fcr, fpr);
- ftp_rqfpr[priv->cur_filer_idx] = fpr;
- ftp_rqfcr[priv->cur_filer_idx] = fcr;
+ priv->ftp_rqfpr[priv->cur_filer_idx] = fpr;
+ priv->ftp_rqfcr[priv->cur_filer_idx] = fcr;
priv->cur_filer_idx = priv->cur_filer_idx - 1;
}
if (ethflow & RXH_IP_SRC) {
fcr = RQFCR_PID_SIA | RQFCR_CMP_NOMATCH | RQFCR_HASH |
RQFCR_AND | RQFCR_HASHTBL_0;
- ftp_rqfpr[priv->cur_filer_idx] = fpr;
- ftp_rqfcr[priv->cur_filer_idx] = fcr;
+ priv->ftp_rqfpr[priv->cur_filer_idx] = fpr;
+ priv->ftp_rqfcr[priv->cur_filer_idx] = fcr;
gfar_write_filer(priv, priv->cur_filer_idx, fcr, fpr);
priv->cur_filer_idx = priv->cur_filer_idx - 1;
}
@@ -643,8 +643,8 @@ static void ethflow_to_filer_rules (struct
gfar_private *priv, u64 ethflow)
if (ethflow & (RXH_IP_DST)) {
fcr = RQFCR_PID_DIA | RQFCR_CMP_NOMATCH | RQFCR_HASH |
RQFCR_AND | RQFCR_HASHTBL_0;
- ftp_rqfpr[priv->cur_filer_idx] = fpr;
- ftp_rqfcr[priv->cur_filer_idx] = fcr;
+ priv->ftp_rqfpr[priv->cur_filer_idx] = fpr;
+ priv->ftp_rqfcr[priv->cur_filer_idx] = fcr;
gfar_write_filer(priv, priv->cur_filer_idx, fcr, fpr);
priv->cur_filer_idx = priv->cur_filer_idx - 1;
}
@@ -652,8 +652,8 @@ static void ethflow_to_filer_rules (struct
gfar_private *priv, u64 ethflow)
if (ethflow & RXH_L3_PROTO) {
fcr = RQFCR_PID_L4P | RQFCR_CMP_NOMATCH | RQFCR_HASH |
RQFCR_AND | RQFCR_HASHTBL_0;
- ftp_rqfpr[priv->cur_filer_idx] = fpr;
- ftp_rqfcr[priv->cur_filer_idx] = fcr;
+ priv->ftp_rqfpr[priv->cur_filer_idx] = fpr;
+ priv->ftp_rqfcr[priv->cur_filer_idx] = fcr;
gfar_write_filer(priv, priv->cur_filer_idx, fcr, fpr);
priv->cur_filer_idx = priv->cur_filer_idx - 1;
}
@@ -661,8 +661,8 @@ static void ethflow_to_filer_rules (struct
gfar_private *priv, u64 ethflow)
if (ethflow & RXH_L4_B_0_1) {
fcr = RQFCR_PID_SPT | RQFCR_CMP_NOMATCH | RQFCR_HASH |
RQFCR_AND | RQFCR_HASHTBL_0;
- ftp_rqfpr[priv->cur_filer_idx] = fpr;
- ftp_rqfcr[priv->cur_filer_idx] = fcr;
+ priv->ftp_rqfpr[priv->cur_filer_idx] = fpr;
+ priv->ftp_rqfcr[priv->cur_filer_idx] = fcr;
gfar_write_filer(priv, priv->cur_filer_idx, fcr, fpr);
priv->cur_filer_idx = priv->cur_filer_idx - 1;
}
@@ -670,8 +670,8 @@ static void ethflow_to_filer_rules (struct
gfar_private *priv, u64 ethflow)
if (ethflow & RXH_L4_B_2_3) {
fcr = RQFCR_PID_DPT | RQFCR_CMP_NOMATCH | RQFCR_HASH |
RQFCR_AND | RQFCR_HASHTBL_0;
- ftp_rqfpr[priv->cur_filer_idx] = fpr;
- ftp_rqfcr[priv->cur_filer_idx] = fcr;
+ priv->ftp_rqfpr[priv->cur_filer_idx] = fpr;
+ priv->ftp_rqfcr[priv->cur_filer_idx] = fcr;
gfar_write_filer(priv, priv->cur_filer_idx, fcr, fpr);
priv->cur_filer_idx = priv->cur_filer_idx - 1;
}
@@ -705,12 +705,12 @@ static int gfar_ethflow_to_filer_table(struct
gfar_private *priv, u64 ethflow, u
}
for (i = 0; i < MAX_FILER_IDX + 1; i++) {
- local_rqfpr[j] = ftp_rqfpr[i];
- local_rqfcr[j] = ftp_rqfcr[i];
+ local_rqfpr[j] = priv->ftp_rqfpr[i];
+ local_rqfcr[j] = priv->ftp_rqfcr[i];
j--;
- if ((ftp_rqfcr[i] == (RQFCR_PID_PARSE |
+ if ((priv->ftp_rqfcr[i] == (RQFCR_PID_PARSE |
RQFCR_CLE |RQFCR_AND)) &&
- (ftp_rqfpr[i] == cmp_rqfpr))
+ (priv->ftp_rqfpr[i] == cmp_rqfpr))
break;
}
@@ -724,20 +724,22 @@ static int gfar_ethflow_to_filer_table(struct
gfar_private *priv, u64 ethflow, u
* if it was already programmed, we need to overwrite these rules
*/
for (l = i+1; l < MAX_FILER_IDX; l++) {
- if ((ftp_rqfcr[l] & RQFCR_CLE) &&
- !(ftp_rqfcr[l] & RQFCR_AND)) {
- ftp_rqfcr[l] = RQFCR_CLE | RQFCR_CMP_EXACT |
+ if ((priv->ftp_rqfcr[l] & RQFCR_CLE) &&
+ !(priv->ftp_rqfcr[l] & RQFCR_AND)) {
+ priv->ftp_rqfcr[l] = RQFCR_CLE | RQFCR_CMP_EXACT |
RQFCR_HASHTBL_0 | RQFCR_PID_MASK;
- ftp_rqfpr[l] = FPR_FILER_MASK;
- gfar_write_filer(priv, l, ftp_rqfcr[l], ftp_rqfpr[l]);
+ priv->ftp_rqfpr[l] = FPR_FILER_MASK;
+ gfar_write_filer(priv, l, priv->ftp_rqfcr[l],
+ priv->ftp_rqfpr[l]);
break;
}
- if (!(ftp_rqfcr[l] & RQFCR_CLE) && (ftp_rqfcr[l] & RQFCR_AND))
+ if (!(priv->ftp_rqfcr[l] & RQFCR_CLE) &&
+ (priv->ftp_rqfcr[l] & RQFCR_AND))
continue;
else {
- local_rqfpr[j] = ftp_rqfpr[l];
- local_rqfcr[j] = ftp_rqfcr[l];
+ local_rqfpr[j] = priv->ftp_rqfpr[l];
+ local_rqfcr[j] = priv->ftp_rqfcr[l];
j--;
}
}
@@ -750,8 +752,8 @@ static int gfar_ethflow_to_filer_table(struct
gfar_private *priv, u64 ethflow, u
/* Write back the popped out rules again */
for (k = j+1; k < MAX_FILER_IDX; k++) {
- ftp_rqfpr[priv->cur_filer_idx] = local_rqfpr[k];
- ftp_rqfcr[priv->cur_filer_idx] = local_rqfcr[k];
+ priv->ftp_rqfpr[priv->cur_filer_idx] = local_rqfpr[k];
+ priv->ftp_rqfcr[priv->cur_filer_idx] = local_rqfcr[k];
gfar_write_filer(priv, priv->cur_filer_idx,
local_rqfcr[k], local_rqfpr[k]);
if (!priv->cur_filer_idx)
--
1.5.6.5
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: rx_ntuple.patch --]
[-- Type: text/x-patch; name="rx_ntuple.patch", Size: 32365 bytes --]
Signed-off-by: Sebastian Poehn <sebastian.poehn@belden.com>
---
drivers/net/gianfar.c | 16 +-
drivers/net/gianfar.h | 47 ++
drivers/net/gianfar_ethtool.c | 969 +++++++++++++++++++++++++++++++++++++++++
3 files changed, 1028 insertions(+), 4 deletions(-)
diff --git a/drivers/net/gianfar.c b/drivers/net/gianfar.c
index ff60b23..ddd4007 100644
--- a/drivers/net/gianfar.c
+++ b/drivers/net/gianfar.c
@@ -658,6 +658,11 @@ static int gfar_of_init(struct platform_device *ofdev, struct net_device **pdev)
priv->num_rx_queues = num_rx_qs;
priv->num_grps = 0x0;
+ /* Init Rx queue filer rule set linked list*/
+ INIT_LIST_HEAD(&priv->ntuple_list.list);
+ priv->ntuple_list.count = 0;
+ mutex_init(&priv->rx_queue_access);
+
model = of_get_property(np, "model", NULL);
for (i = 0; i < MAXGROUPS; i++)
@@ -751,7 +756,8 @@ static int gfar_of_init(struct platform_device *ofdev, struct net_device **pdev)
FSL_GIANFAR_DEV_HAS_VLAN |
FSL_GIANFAR_DEV_HAS_MAGIC_PACKET |
FSL_GIANFAR_DEV_HAS_EXTENDED_HASH |
- FSL_GIANFAR_DEV_HAS_TIMER;
+ FSL_GIANFAR_DEV_HAS_TIMER |
+ FSL_GIANFAR_DEV_HAS_RX_FILER;
ctype = of_get_property(np, "phy-connection-type", NULL);
@@ -1042,6 +1048,9 @@ static int gfar_probe(struct platform_device *ofdev)
if (priv->device_flags & FSL_GIANFAR_DEV_HAS_VLAN)
dev->features |= NETIF_F_HW_VLAN_TX | NETIF_F_HW_VLAN_RX;
+ if (priv->device_flags & FSL_GIANFAR_DEV_HAS_RX_FILER)
+ dev->features |= NETIF_F_NTUPLE;
+
if (priv->device_flags & FSL_GIANFAR_DEV_HAS_EXTENDED_HASH) {
priv->extended_hash = 1;
priv->hash_width = 9;
@@ -1151,9 +1160,8 @@ static int gfar_probe(struct platform_device *ofdev)
priv->rx_queue[i]->rxic = DEFAULT_RXIC;
}
- /* enable filer if using multiple RX queues*/
- if(priv->num_rx_queues > 1)
- priv->rx_filer_enable = 1;
+ /* always enable rx filer*/
+ priv->rx_filer_enable = 1;
/* Enable most messages by default */
priv->msg_enable = (NETIF_MSG_IFUP << 1 ) - 1;
diff --git a/drivers/net/gianfar.h b/drivers/net/gianfar.h
index fc86f51..6c94e7b 100644
--- a/drivers/net/gianfar.h
+++ b/drivers/net/gianfar.h
@@ -168,6 +168,7 @@ extern const char gfar_driver_version[];
#define MACCFG2_LENGTHCHECK 0x00000010
#define MACCFG2_MPEN 0x00000008
+#define ECNTRL_FIFM 0x00008000
#define ECNTRL_INIT_SETTINGS 0x00001000
#define ECNTRL_TBI_MODE 0x00000020
#define ECNTRL_REDUCED_MODE 0x00000010
@@ -271,6 +272,7 @@ extern const char gfar_driver_version[];
#define RCTRL_TUCSEN 0x00000100
#define RCTRL_PRSDEP_MASK 0x000000c0
#define RCTRL_PRSDEP_INIT 0x000000c0
+#define RCTRL_PRSFM 0x00000020
#define RCTRL_PROM 0x00000008
#define RCTRL_EMEN 0x00000002
#define RCTRL_REQ_PARSER (RCTRL_VLEX | RCTRL_IPCSEN | \
@@ -870,6 +872,7 @@ struct gfar {
#define FSL_GIANFAR_DEV_HAS_BD_STASHING 0x00000200
#define FSL_GIANFAR_DEV_HAS_BUF_STASHING 0x00000400
#define FSL_GIANFAR_DEV_HAS_TIMER 0x00000800
+#define FSL_GIANFAR_DEV_HAS_RX_FILER 0x00001000
#if (MAXGROUPS == 2)
#define DEFAULT_MAPPING 0xAA
@@ -1066,6 +1069,9 @@ struct gfar_private {
struct vlan_group *vlgrp;
+ /* RX queue filer rule set*/
+ struct ethtool_rx_ntuple_list ntuple_list;
+ struct mutex rx_queue_access;
/* Hash registers and their width */
u32 __iomem *hash_regs[16];
@@ -1140,6 +1146,16 @@ static inline void gfar_write_filer(struct gfar_private *priv,
gfar_write(®s->rqfpr, fpr);
}
+static inline void gfar_read_filer(struct gfar_private *priv,
+ unsigned int far, unsigned int *fcr, unsigned int *fpr)
+{
+ struct gfar __iomem *regs = priv->gfargrp[0].regs;
+
+ gfar_write(®s->rqfar, far);
+ *fcr = gfar_read(®s->rqfcr);
+ *fpr = gfar_read(®s->rqfpr);
+}
+
extern void lock_rx_qs(struct gfar_private *priv);
extern void lock_tx_qs(struct gfar_private *priv);
extern void unlock_rx_qs(struct gfar_private *priv);
@@ -1157,4 +1173,35 @@ int gfar_set_features(struct net_device *dev, u32 features);
extern const struct ethtool_ops gfar_ethtool_ops;
+#define ESWFULL 160
+#define EHWFULL 161
+#define EOUTOFRANGE 162
+#define MAX_FILER_CACHE_IDX (2*(MAX_FILER_IDX))
+
+#define RQFCR_PID_PRI_MASK 0xFFFFFFF8
+#define RQFCR_PID_L4P_MASK 0xFFFFFF00
+#define RQFCR_PID_VID_MASK 0xFFFFF000
+#define RQFCR_PID_PORT_MASK 0xFFFF0000
+#define RQFCR_PID_MAC_MASK 0xFF000000
+
+struct gfar_mask_entry {
+ unsigned int mask; /*The mask value which is valid for a block with*/
+ unsigned int start; /*start*/
+ unsigned int end; /*till end*/
+ unsigned int block; /*Same block values indicate depended entries*/
+};
+
+/*Represents a receive filer table entry */
+struct gfar_filer_entry {
+ u32 ctrl; /*The control field from HW*/
+ u32 prop; /*The property field from HW*/
+};
+
+
+/*The 20 additional entries are a shadow for one extra element*/
+struct filer_table {
+ u32 index;
+ struct gfar_filer_entry fe[MAX_FILER_CACHE_IDX + 20];
+};
+
#endif /* __GIANFAR_H */
diff --git a/drivers/net/gianfar_ethtool.c b/drivers/net/gianfar_ethtool.c
index 493d743..e0dcc1c 100644
--- a/drivers/net/gianfar_ethtool.c
+++ b/drivers/net/gianfar_ethtool.c
@@ -37,6 +37,7 @@
#include <linux/ethtool.h>
#include <linux/mii.h>
#include <linux/phy.h>
+#include <linux/sort.h>
#include "gianfar.h"
@@ -787,6 +788,973 @@ static int gfar_set_nfc(struct net_device *dev, struct ethtool_rxnfc *cmd)
return ret;
}
+static int gfar_check_filer_hardware(struct gfar_private *priv)
+{
+ struct gfar __iomem *regs = NULL;
+ u32 i;
+
+ regs = priv->gfargrp[0].regs;
+
+ /*Check if we are in FIFO mode*/
+ i = gfar_read(®s->ecntrl);
+ i &= ECNTRL_FIFM;
+ if (i == ECNTRL_FIFM) {
+ printk(KERN_WARNING "Interface in FIFO mode\n");
+ i = gfar_read(®s->rctrl);
+ i &= RCTRL_PRSDEP_MASK | RCTRL_PRSFM;
+ if (i == (RCTRL_PRSDEP_MASK | RCTRL_PRSFM)) {
+ printk(KERN_WARNING
+ "Receive Queue Filtering is enabled\n");
+ } else {
+ printk(KERN_WARNING
+ "Receive Queue Filtering is disabled\n");
+ return -1;
+ }
+ }
+ /*Or in standard mode*/
+ else{
+ i = gfar_read(®s->rctrl);
+ i &= RCTRL_PRSDEP_MASK;
+ if (i == RCTRL_PRSDEP_MASK) {
+ printk(KERN_WARNING
+ "Receive Queue Filtering is enabled\n");
+ } else {
+ printk(KERN_WARNING
+ "Receive Queue Filtering is disabled\n");
+ return -1;
+ }
+ }
+
+ /* Sets the properties for arbitrary filer rule
+ * to the first 4 Layer 4 Bytes*/
+ regs->rbifx = 0xC0C1C2C3;
+ return 0;
+}
+
+static int gfar_comp_asc(const void *a, const void *b)
+{
+ if (*(u32 *) a > *(u32 *) b)
+ return 1;
+ else if (*(u32 *) a == *(u32 *) b)
+ return 0;
+ else
+ return -1;
+}
+
+static int gfar_comp_desc(const void *a, const void *b)
+{
+ if (*(u32 *) a > *(u32 *) b)
+ return -1;
+ else if (*(u32 *) a == *(u32 *) b)
+ return 0;
+ else
+ return 1;
+}
+
+static void gfar_swap(void *a, void *b, int size)
+{
+ u32 t1 = *(u32 *) a;
+ u32 t2 = *(u32 *) (a + 4);
+ u32 t3 = *(u32 *) (a + 8);
+ u32 t4 = *(u32 *) (a + 12);
+ *(u32 *) a = *(u32 *) b;
+ *(u32 *) (a + 4) = *(u32 *) (b + 4);
+ *(u32 *) (a + 8) = *(u32 *) (b + 8);
+ *(u32 *) (a + 12) = *(u32 *) (b + 12);
+ *(u32 *) b = t1;
+ *(u32 *) (b + 4) = t2;
+ *(u32 *) (b + 8) = t3;
+ *(u32 *) (b + 12) = t4;
+}
+
+/*Write a mask to filer cache*/
+static void gfar_set_mask(u32 mask, struct filer_table *tab)
+{
+ tab->fe[tab->index].ctrl = RQFCR_AND | RQFCR_PID_MASK
+ | RQFCR_CMP_EXACT;
+ tab->fe[tab->index].prop = mask;
+ tab->index++;
+}
+
+/*Sets parse bits (e.g. IP or TCP)*/
+static void gfar_set_parse_bits(u32 value, u32 mask, struct filer_table *tab)
+{
+ gfar_set_mask(mask, tab);
+ tab->fe[tab->index].ctrl = RQFCR_CMP_EXACT | RQFCR_PID_PARSE
+ | RQFCR_AND;
+ tab->fe[tab->index].prop = value;
+ tab->index++;
+}
+
+static void gfar_set_general_attribute(u32 value,
+ u32 mask, u32 flag, struct filer_table *tab)
+{
+ gfar_set_mask(mask, tab);
+ tab->fe[tab->index].ctrl = RQFCR_CMP_EXACT | RQFCR_AND
+ | flag;
+ tab->fe[tab->index].prop = value;
+ tab->index++;
+}
+
+/*
+ * For setting a tuple of value and mask of type flag
+ * Example:
+ * IP-Src = 10.0.0.0/255.0.0.0
+ * value: 0x0A000000 mask: FF000000 flag: RQFPR_IPV4
+ *
+ * Ethtool gives us a value=0 and mask=~0 for don't care a tuple
+ * For a don't care mask it gives us a 0
+ *
+ * The check if don't care and the mask adjustment if mask=0 is done for VLAN
+ * and MAC stuff on an upper level (due to missing information on this level).
+ * For these guys we can discard them if they are value=0 and mask=0.
+ *
+ * Further the all masks are one-padded for better hardware efficiency.
+ */
+static void gfar_set_attribute(u32 value, u32 mask,
+ u32 flag, struct filer_table *tab)
+{
+ switch (flag) {
+ /*3bit*/
+ case RQFCR_PID_PRI:
+ if (!(value | mask))
+ return;
+ mask |= RQFCR_PID_PRI_MASK;
+ break;
+ /*8bit*/
+ case RQFCR_PID_L4P:
+ case RQFCR_PID_TOS:
+ if (!~(mask|RQFCR_PID_L4P_MASK))
+ return;
+ if (!mask)
+ mask = ~0;
+ else
+ mask |= RQFCR_PID_L4P_MASK;
+ break;
+ /*12bit*/
+ case RQFCR_PID_VID:
+ if (!(value | mask))
+ return;
+ mask |= RQFCR_PID_VID_MASK;
+ break;
+ /*16bit*/
+ case RQFCR_PID_DPT:
+ case RQFCR_PID_SPT:
+ case RQFCR_PID_ETY:
+ if (!~(mask | RQFCR_PID_PORT_MASK))
+ return;
+ if (!mask)
+ mask = ~0;
+ else
+ mask |= RQFCR_PID_PORT_MASK;
+ break;
+ /*24bit*/
+ case RQFCR_PID_DAH:
+ case RQFCR_PID_DAL:
+ case RQFCR_PID_SAH:
+ case RQFCR_PID_SAL:
+ if (!(value | mask))
+ return;
+ mask |= RQFCR_PID_MAC_MASK;
+ break;
+ /*for all real 32bit masks*/
+ default:
+ if (!~mask)
+ return;
+ if (!mask)
+ mask = ~0;
+ break;
+ }
+ gfar_set_general_attribute(value, mask, flag, tab);
+}
+
+/*Translates value and mask for UDP, TCP or SCTP*/
+static void gfar_set_basic_ip(struct ethtool_tcpip4_spec *value,
+ struct ethtool_tcpip4_spec *mask, struct filer_table *tab)
+{
+ gfar_set_attribute(value->ip4src, mask->ip4src, RQFCR_PID_SIA, tab);
+ gfar_set_attribute(value->ip4dst, mask->ip4dst, RQFCR_PID_DIA, tab);
+ gfar_set_attribute(value->pdst, mask->pdst, RQFCR_PID_DPT, tab);
+ gfar_set_attribute(value->psrc, mask->psrc, RQFCR_PID_SPT, tab);
+ gfar_set_attribute(value->tos, mask->tos, RQFCR_PID_TOS, tab);
+}
+
+/*Translates value and mask for RAW-IP4*/
+static void gfar_set_user_ip(struct ethtool_usrip4_spec *value,
+ struct ethtool_usrip4_spec *mask, struct filer_table *tab)
+{
+ gfar_set_attribute(value->ip4src, mask->ip4src, RQFCR_PID_SIA, tab);
+ gfar_set_attribute(value->ip4dst, mask->ip4dst, RQFCR_PID_DIA, tab);
+ gfar_set_attribute(value->tos, mask->tos, RQFCR_PID_TOS, tab);
+ gfar_set_attribute(value->proto, mask->proto, RQFCR_PID_L4P, tab);
+ gfar_set_attribute(value->l4_4_bytes, mask->l4_4_bytes, RQFCR_PID_ARB,
+ tab);
+
+}
+
+/*Translates value and mask for ETHER spec*/
+static void gfar_set_ether(struct ethhdr *value, struct ethhdr *mask,
+ struct filer_table *tab)
+{
+ u32 upper_temp_mask = 0;
+ u32 lower_temp_mask = 0;
+ /*Source address*/
+ if (!is_broadcast_ether_addr(mask->h_source)) {
+
+ if (is_zero_ether_addr(mask->h_source)) {
+ upper_temp_mask = 0xFFFFFFFF;
+ lower_temp_mask = 0xFFFFFFFF;
+ } else {
+ upper_temp_mask = mask->h_source[0] << 16
+ | mask->h_source[1] << 8
+ | mask->h_source[2] ;
+ lower_temp_mask = mask->h_source[3] << 16
+ | mask->h_source[4] << 8
+ | mask->h_source[5] ;
+ }
+ /*Upper 24bit*/
+ gfar_set_attribute(value->h_source[0] << 16
+ | value->h_source[1] << 8
+ | value->h_source[2],
+ upper_temp_mask, RQFCR_PID_SAH, tab);
+ /*And the same for the lower part*/
+ gfar_set_attribute(value->h_source[3] << 16
+ | value->h_source[4] << 8
+ | value->h_source[5],
+ lower_temp_mask, RQFCR_PID_SAL, tab);
+ }
+ /*Destination address*/
+ if (!is_broadcast_ether_addr(mask->h_dest)) {
+
+ /*Special for destination is limited broadcast*/
+ if ((is_broadcast_ether_addr(value->h_dest)
+ && is_zero_ether_addr(mask->h_dest))) {
+ gfar_set_parse_bits(RQFPR_EBC, RQFPR_EBC, tab);
+ } else {
+
+ if (is_zero_ether_addr(mask->h_dest)) {
+ upper_temp_mask = 0xFFFFFFFF;
+ lower_temp_mask = 0xFFFFFFFF;
+ } else {
+ upper_temp_mask = mask->h_dest[0] << 16
+ | mask->h_dest[1] << 8
+ | mask->h_dest[2] ;
+ lower_temp_mask = mask->h_dest[3] << 16
+ | mask->h_dest[4] << 8
+ | mask->h_dest[5] ;
+ }
+
+ /*Upper 24bit*/
+ gfar_set_attribute(value->h_dest[0] << 16
+ | value->h_dest[1] << 8
+ | value->h_dest[2],
+ upper_temp_mask, RQFCR_PID_DAH, tab);
+ /*And the same for the lower part*/
+ gfar_set_attribute(value->h_dest[3] << 16
+ | value->h_dest[4] << 8
+ | value->h_dest[5],
+ lower_temp_mask, RQFCR_PID_DAL, tab);
+ }
+ }
+
+ gfar_set_attribute(value->h_proto, mask->h_proto, RQFCR_PID_ETY, tab);
+
+}
+
+/*Convert a ethtool_rx_ntuple to binary filter format of gianfar*/
+static int gfar_convert_to_filer(struct ethtool_rx_ntuple_flow_spec *rule,
+ struct filer_table *tab)
+{
+ u32 vlan = 0, vlan_mask = 0;
+ u32 id = 0, id_mask = 0;
+ u32 cfi = 0, cfi_mask = 0;
+ u32 prio = 0, prio_mask = 0;
+
+ u32 old_index = tab->index;
+
+ /*Check if vlan is wanted*/
+ if (rule->vlan_tag_mask != 0xFFFF) {
+ if (!rule->vlan_tag_mask)
+ rule->vlan_tag_mask = 0xFFFF;
+
+ vlan = RQFPR_VLN;
+ vlan_mask = RQFPR_VLN;
+
+ /*Separate the fields*/
+ id = rule->vlan_tag & 0xFFF;
+ id_mask = rule->vlan_tag_mask & 0xFFF;
+ cfi = (rule->vlan_tag >> 12) & 1;
+ cfi_mask = (rule->vlan_tag_mask >> 12) & 1;
+ prio = (rule->vlan_tag >> 13) & 0x7;
+ prio_mask = (rule->vlan_tag_mask >> 13) & 0x7;
+
+ if (cfi == 1 && cfi_mask == 1) {
+ vlan |= RQFPR_CFI;
+ vlan_mask |= RQFPR_CFI;
+ } else if (cfi == 0 && cfi_mask == 1) {
+ vlan_mask |= RQFPR_CFI;
+ }
+ }
+
+ switch (rule->flow_type) {
+ case TCP_V4_FLOW:
+ gfar_set_parse_bits(RQFPR_IPV4 | RQFPR_TCP | vlan, RQFPR_IPV4
+ | RQFPR_TCP | vlan_mask, tab);
+ gfar_set_basic_ip((struct ethtool_tcpip4_spec *) &rule->h_u,
+ (struct ethtool_tcpip4_spec *) &rule->m_u, tab);
+
+ break;
+ case UDP_V4_FLOW:
+ gfar_set_parse_bits(RQFPR_IPV4 | RQFPR_UDP | vlan, RQFPR_IPV4
+ | RQFPR_UDP | vlan_mask, tab);
+ gfar_set_basic_ip((struct ethtool_tcpip4_spec *) &rule->h_u,
+ (struct ethtool_tcpip4_spec *) &rule->m_u, tab);
+ break;
+ case SCTP_V4_FLOW:
+ gfar_set_parse_bits(RQFPR_IPV4 | vlan, RQFPR_IPV4 | vlan_mask,
+ tab);
+ gfar_set_attribute(132, 0xFFFFFFFF, RQFCR_PID_L4P, tab);
+ gfar_set_basic_ip((struct ethtool_tcpip4_spec *) &rule->h_u,
+ (struct ethtool_tcpip4_spec *) &rule->m_u, tab);
+ break;
+ case IP_USER_FLOW:
+ gfar_set_parse_bits(RQFPR_IPV4 | vlan, RQFPR_IPV4 | vlan_mask,
+ tab);
+ gfar_set_user_ip((struct ethtool_usrip4_spec *) &rule->h_u,
+ (struct ethtool_usrip4_spec *) &rule->m_u, tab);
+ break;
+ case ETHER_FLOW:
+ if (vlan)
+ gfar_set_parse_bits(vlan, vlan_mask, tab);
+ gfar_set_ether((struct ethhdr *) &rule->h_u,
+ (struct ethhdr *) &rule->m_u, tab);
+ break;
+ default:
+ return -1;
+ }
+
+ /*Set the vlan attributes in the end*/
+ if (vlan) {
+ gfar_set_attribute(id, id_mask, RQFCR_PID_VID, tab);
+ gfar_set_attribute(prio, prio_mask, RQFCR_PID_PRI, tab);
+ }
+
+ /*If there has been nothing written till now, it must be a default*/
+ if (tab->index == old_index) {
+ gfar_set_mask(0xFFFFFFFF, tab);
+ tab->fe[tab->index].ctrl = 0x20;
+ tab->fe[tab->index].prop = 0x0;
+ tab->index++;
+ }
+
+ /*Remove last AND*/
+ tab->fe[tab->index - 1].ctrl &= (~RQFCR_AND);
+
+ /*Specify which queue to use or to drop*/
+ if (rule->action == ETHTOOL_RXNTUPLE_ACTION_DROP)
+ tab->fe[tab->index - 1].ctrl |= RQFCR_RJE;
+ else
+ tab->fe[tab->index - 1].ctrl |= (rule->action << 10);
+
+ /*Only big enough entries can be clustered*/
+ if (tab->index > (old_index + 2)) {
+ tab->fe[old_index + 1].ctrl |= RQFCR_CLE;
+ tab->fe[tab->index - 1].ctrl |= RQFCR_CLE;
+ }
+
+ /*In rare cases the cache can be full while there is free space in hw*/
+ if (tab->index > MAX_FILER_CACHE_IDX - 1)
+ return -ESWFULL;
+
+ return 0;
+}
+
+/*Copy size filer entries*/
+static void gfar_copy_filer_entries(struct gfar_filer_entry dst[0],
+ struct gfar_filer_entry src[0], s32 size)
+{
+ while (size > 0) {
+ size--;
+ dst[size].ctrl = src[size].ctrl;
+ dst[size].prop = src[size].prop;
+ }
+}
+
+/* Delete the contents of the filer-table between start and end
+ * and collapse them*/
+static int gfar_trim_filer_entries(u32 begin, u32 end, struct filer_table *tab)
+{
+ int length;
+ if (end > MAX_FILER_CACHE_IDX || end < begin)
+ return -EOUTOFRANGE;
+
+ end++;
+ length = end - begin;
+
+ /*Copy*/
+ while (end < tab->index) {
+ tab->fe[begin].ctrl = tab->fe[end].ctrl;
+ tab->fe[begin++].prop = tab->fe[end++].prop;
+
+ }
+ /*Fill up with don't cares*/
+ while (begin <= tab->index) {
+ tab->fe[begin].ctrl = 0x60;
+ tab->fe[begin].prop = 0xFFFFFFFF;
+ begin++;
+ }
+
+ tab->index -= length;
+ return 0;
+}
+
+/*Make space on the wanted location*/
+static int gfar_expand_filer_entries(u32 begin, u32 length,
+ struct filer_table *tab)
+{
+ int i = 0;
+ if (length || length + tab->index
+ > MAX_FILER_CACHE_IDX || begin > MAX_FILER_CACHE_IDX)
+ return -EOUTOFRANGE;
+
+ gfar_copy_filer_entries(&(tab->fe[begin + length]),
+ &(tab->fe[begin]),
+ tab->index - length + 1);
+
+ tab->index += length;
+ return 0;
+}
+
+static int gfar_get_next_cluster_start(int start, struct filer_table *tab)
+{
+ for (; (start < tab->index) && (start < MAX_FILER_CACHE_IDX - 1);
+ start++) {
+ if ((tab->fe[start].ctrl & (RQFCR_AND | RQFCR_CLE))
+ == (RQFCR_AND | RQFCR_CLE)) {
+ return start;
+ }
+ }
+ return -1;
+}
+
+static int gfar_get_next_cluster_end(int start, struct filer_table *tab)
+{
+ for (; (start < tab->index) && (start < MAX_FILER_CACHE_IDX - 1);
+ start++) {
+ if ((tab->fe[start].ctrl & (RQFCR_AND | RQFCR_CLE))
+ == (RQFCR_CLE))
+ return start;
+ }
+ return -1;
+}
+
+/*
+ * Uses hardwares clustering option to reduce
+ * the number of filer table entries
+ */
+static void gfar_cluster_filer(struct filer_table *tab)
+{
+ s32 i = -1, j, iend, jend;
+
+ while ((i = gfar_get_next_cluster_start(++i, tab)) != -1) {
+ j = i;
+ while ((j = gfar_get_next_cluster_start(++j, tab)) != -1) {
+ /*
+ * The cluster entries self and the previous one
+ * (a mask) must be identical!
+ */
+ if (tab->fe[i].ctrl != tab->fe[j].ctrl)
+ break;
+ if (tab->fe[i].prop != tab->fe[j].prop)
+ break;
+ if (tab->fe[i - 1].ctrl != tab->fe[j - 1].ctrl)
+ break;
+ if (tab->fe[i - 1].prop != tab->fe[j - 1].prop)
+ break;
+ iend = gfar_get_next_cluster_end(i, tab);
+ jend = gfar_get_next_cluster_end(j, tab);
+ if (jend == -1 || iend == -1)
+ break;
+ /*
+ * First we make some free space, where our cluster
+ * element should be. Then we copy it there and finally
+ * delete in from its old location.
+ */
+
+ if (gfar_expand_filer_entries(iend, (jend - j), tab)
+ == -EOUTOFRANGE)
+ break;
+
+ gfar_copy_filer_entries(&(tab->fe[iend + 1]),
+ &(tab->fe[jend + 1]), jend - j);
+
+ if (gfar_trim_filer_entries(jend - 1, jend + (jend - j),
+ tab) == -EOUTOFRANGE)
+ return;
+
+ /*Mask out cluster bit*/
+ tab->fe[iend].ctrl &= ~(RQFCR_CLE);
+ }
+ }
+}
+
+/*Swaps the 0xFF80 masked bits of a1<>a2 and b1<>b2*/
+static void gfar_swap_ff80_bits(struct gfar_filer_entry *a1,
+ struct gfar_filer_entry *a2,
+ struct gfar_filer_entry *b1,
+ struct gfar_filer_entry *b2)
+{
+ u32 temp[4];
+ temp[0] = a1->ctrl & 0xFF80;
+ temp[1] = a2->ctrl & 0xFF80;
+ temp[2] = b1->ctrl & 0xFF80;
+ temp[3] = b2->ctrl & 0xFF80;
+
+ a1->ctrl &= ~0xFF80;
+ a2->ctrl &= ~0xFF80;
+ b1->ctrl &= ~0xFF80;
+ b2->ctrl &= ~0xFF80;
+
+ a1->ctrl |= temp[1];
+ a2->ctrl |= temp[0];
+ b1->ctrl |= temp[3];
+ b2->ctrl |= temp[2];
+}
+
+/* Generate a list consisting of masks values with their start and
+ * end of validity and block as indicator for parts belonging
+ * together (glued by ANDs) in mask_table*/
+u32 gfar_generate_mask_table(struct gfar_mask_entry *mask_table,
+ struct filer_table *tab)
+{
+ u32 i, and_index = 0, block_index = 1;
+
+ for (i = 0; i < tab->index; i++) {
+
+ /*LSByte of control = 0 sets a mask*/
+ if (!(tab->fe[i].ctrl & 0xF)) {
+ mask_table[and_index].mask = tab->fe[i].prop;
+ mask_table[and_index].start = i;
+ mask_table[and_index].block = block_index;
+ if (and_index >= 1)
+ mask_table[and_index - 1].end = i - 1;
+ and_index++;
+ }
+ /*cluster starts will be separated because they should
+ * hold their position*/
+ if (tab->fe[i].ctrl & RQFCR_CLE)
+ block_index++;
+ /*A not set AND indicated the end of a depended block*/
+ if (!(tab->fe[i].ctrl & RQFCR_AND))
+ block_index++;
+
+ }
+
+ mask_table[and_index - 1].end = i - 1;
+
+ return and_index;
+}
+
+/*
+* Sorts the entries of mask_table by the values of the masks.
+* Important: The 0xFF80 flags of the first and last entry of a
+* block must hold their position (which queue, CLusterEnable, ReJEct,
+* AND)
+*/
+void gfar_sort_mask_table(struct gfar_mask_entry *mask_table,
+ struct filer_table *temp_table, u32 and_index)
+{
+ /*Pointer to compare function (_asc or _desc)*/
+ int (*gfar_comp) (const void *, const void *);
+
+ u32 i, size = 0, start = 0, prev = 1;
+ u32 old_first, old_last, new_first, new_last;
+
+ gfar_comp = &gfar_comp_desc;
+
+ for (i = 0; i < and_index; i++) {
+
+ if (prev != mask_table[i].block) {
+ old_first = mask_table[start].start + 1;
+ old_last = mask_table[i - 1].end;
+ /*I my opinion start should be multiplied by
+ * sizeof(struct gfar_mask_entry) do not ask me why
+ * only this version is working */
+ sort(mask_table + start, size,
+ sizeof(struct gfar_mask_entry),
+ gfar_comp, &gfar_swap);
+
+ /*Toggle order for every block. This makes the
+ * thing more efficient!*/
+ if (gfar_comp == gfar_comp_desc)
+ gfar_comp = &gfar_comp_asc;
+ else
+ gfar_comp = &gfar_comp_desc;
+
+ new_first = mask_table[start].start + 1;
+ new_last = mask_table[i - 1].end;
+
+ gfar_swap_ff80_bits(&temp_table->fe[new_first],
+ &temp_table->fe[old_first],
+ &temp_table->fe[new_last],
+ &temp_table->fe[old_last]);
+
+ start = i;
+ size = 0;
+ }
+ size++;
+ prev = mask_table[i].block;
+ }
+}
+
+/*
+ * Reduces the number of masks needed in the filer table to save entries
+ * This is done by sorting the masks of a depended block. A depended block is
+ * identified by gluing ANDs or CLE. The sorting order toggles after every
+ * block. Of course entries in scope of a mask must change their location with
+ * it.
+*/
+static int gfar_optimize_filer_masks(struct filer_table *tab)
+{
+ struct filer_table *temp_table;
+ struct gfar_mask_entry *mask_table;
+
+ u32 and_index = 0, previous_mask = 0, i = 0, j = 0, size = 0;
+ s32 ret = 0;
+
+ /*We need a copy of the filer table because
+ * we want to change its order*/
+ temp_table = kmalloc(sizeof(struct filer_table), GFP_KERNEL);
+ if (temp_table == NULL)
+ return -ENOMEM;
+ memcpy(temp_table, tab, sizeof(struct filer_table));
+
+ mask_table = kzalloc(sizeof(struct gfar_mask_entry)
+ * (MAX_FILER_CACHE_IDX / 2 + 1), GFP_KERNEL);
+ if (mask_table == NULL) {
+ ret = -ENOMEM;
+ goto end;
+ }
+
+ and_index = gfar_generate_mask_table(mask_table, tab);
+
+ gfar_sort_mask_table(mask_table, temp_table, and_index);
+
+ /*Now we can copy the data from our duplicated filer table to
+ * the real one in the order the mask table says*/
+ for (i = 0; i < and_index; i++) {
+ size = mask_table[i].end - mask_table[i].start + 1;
+ gfar_copy_filer_entries(&(tab->fe[j]),
+ &(temp_table->fe[mask_table[i].start]), size);
+ j += size;
+ }
+
+ /* And finally we just have to check for duplicated masks and drop the
+ * second ones*/
+ for (i = 0; i < tab->index && i < MAX_FILER_CACHE_IDX; i++) {
+ if (tab->fe[i].ctrl == 0x80) {
+ previous_mask = i++;
+ break;
+ }
+ }
+ for (; i < tab->index && i < MAX_FILER_CACHE_IDX; i++) {
+ if (tab->fe[i].ctrl == 0x80) {
+ if (tab->fe[i].prop == tab->fe[previous_mask].prop) {
+ /*Two identical ones found!
+ * So drop the second one!*/
+ gfar_trim_filer_entries(i, i, tab);
+
+ } else
+ /*Not identical!*/
+ previous_mask = i;
+ }
+ }
+
+ kfree(mask_table);
+end: kfree(temp_table);
+ return ret;
+}
+
+/*Write the bit-pattern from software's buffer to hardware registers*/
+static int gfar_write_filer_table(struct gfar_private *priv,
+ struct filer_table *tab)
+{
+ u32 i = 0;
+ if (tab->index > MAX_FILER_IDX - 1)
+ return -EHWFULL;
+
+ /*Avoid inconsistent filer table to be processed*/
+ lock_rx_qs(priv);
+
+ /*Fill regular entries*/
+ for (; i < MAX_FILER_IDX - 1 &&
+ (tab->fe[i].ctrl | tab->fe[i].ctrl) ; i++)
+ gfar_write_filer(priv, i, tab->fe[i].ctrl,
+ tab->fe[i].prop);
+ /*Fill the rest with fall-troughs*/
+ for (; i < MAX_FILER_IDX - 1; i++)
+ gfar_write_filer(priv, i, 0x60, 0xFFFFFFFF);
+ /* Last entry must be default accept
+ * because that's what people expect*/
+ gfar_write_filer(priv, i, 0x20, 0x0);
+
+ unlock_rx_qs(priv);
+
+ return 0;
+}
+
+static int gfar_add_table_entry(struct ethtool_rx_ntuple_flow_spec *flow,
+ struct ethtool_rx_ntuple_list *list)
+{
+ struct ethtool_rx_ntuple_flow_spec_container *temp;
+ temp = kmalloc(sizeof(struct ethtool_rx_ntuple_flow_spec_container),
+ GFP_KERNEL);
+ if (temp == NULL)
+ return -ENOMEM;
+ memcpy(&temp->fs, flow, sizeof(struct ethtool_rx_ntuple_flow_spec));
+ list_add_tail(&temp->list, &list->list);
+ list->count++;
+
+ if (~flow->data_mask)
+ printk(KERN_WARNING
+ "User-specific data is not supported by hardware!\n");
+ if (flow->flow_type == IP_USER_FLOW)
+ if (flow->m_u.usr_ip4_spec.ip_ver != 255)
+ printk(KERN_WARNING
+ "IP-Version is not supported by hardware!\n");
+
+ return 0;
+}
+
+/*
+ * Compares flow-specs a and b
+ * if a==b return 0
+ * if a!=b return 1
+ * if error return -1
+ */
+static int gfar_compare_flow_spec(struct ethtool_rx_ntuple_flow_spec *a,
+ struct ethtool_rx_ntuple_flow_spec *b)
+{
+ if (a == 0 || b == 0)
+ return -1;
+ if (a->flow_type != b->flow_type)
+ return 1;
+ if (a->vlan_tag != b->vlan_tag)
+ return 1;
+ if (a->vlan_tag_mask != b->vlan_tag_mask)
+ return 1;
+ switch (a->flow_type) {
+ case TCP_V4_FLOW:
+ case UDP_V4_FLOW:
+ case SCTP_V4_FLOW:
+ if (a->h_u.tcp_ip4_spec.pdst != b->h_u.tcp_ip4_spec.pdst)
+ return 1;
+ if (a->m_u.tcp_ip4_spec.pdst != b->m_u.tcp_ip4_spec.pdst)
+ return 1;
+ if (a->h_u.tcp_ip4_spec.psrc != b->h_u.tcp_ip4_spec.psrc)
+ return 1;
+ if (a->m_u.tcp_ip4_spec.psrc != b->m_u.tcp_ip4_spec.psrc)
+ return 1;
+
+ goto gfar_compare_basic_ip_stuff;
+ case IP_USER_FLOW:
+ if (a->h_u.usr_ip4_spec.proto != b->h_u.usr_ip4_spec.proto)
+ return 1;
+ if (a->m_u.usr_ip4_spec.proto != b->m_u.usr_ip4_spec.proto)
+ return 1;
+ if (a->h_u.usr_ip4_spec.ip_ver != b->h_u.usr_ip4_spec.ip_ver)
+ return 1;
+ if (a->m_u.usr_ip4_spec.ip_ver != b->m_u.usr_ip4_spec.ip_ver)
+ return 1;
+ if (a->h_u.usr_ip4_spec.l4_4_bytes
+ != b->h_u.usr_ip4_spec.l4_4_bytes)
+ return 1;
+ if (a->m_u.usr_ip4_spec.l4_4_bytes
+ != b->m_u.usr_ip4_spec.l4_4_bytes)
+ return 1;
+
+ goto gfar_compare_basic_ip_stuff;
+ case ETHER_FLOW:
+ if (compare_ether_addr(a->h_u.ether_spec.h_dest,
+ b->h_u.ether_spec.h_dest))
+ return 1;
+ if (compare_ether_addr(a->h_u.ether_spec.h_source,
+ b->h_u.ether_spec.h_source))
+ return 1;
+ if (compare_ether_addr(a->m_u.ether_spec.h_dest,
+ b->m_u.ether_spec.h_dest))
+ return 1;
+ if (compare_ether_addr(a->m_u.ether_spec.h_source,
+ b->m_u.ether_spec.h_source))
+ return 1;
+ if (a->h_u.ether_spec.h_proto != b->h_u.ether_spec.h_proto)
+ return 1;
+ if (a->m_u.ether_spec.h_proto != b->m_u.ether_spec.h_proto)
+ return 1;
+ return 0;
+ default:
+ return -1;
+ }
+
+ /*Control-flow never passes here!*/
+
+gfar_compare_basic_ip_stuff:
+ if (a->h_u.tcp_ip4_spec.ip4dst != b->h_u.tcp_ip4_spec.ip4dst)
+ return 1;
+ if (a->m_u.tcp_ip4_spec.ip4dst != b->m_u.tcp_ip4_spec.ip4dst)
+ return 1;
+ if (a->h_u.tcp_ip4_spec.ip4src != b->h_u.tcp_ip4_spec.ip4src)
+ return 1;
+ if (a->m_u.tcp_ip4_spec.ip4src != b->m_u.tcp_ip4_spec.ip4src)
+ return 1;
+ if (a->h_u.tcp_ip4_spec.tos != b->h_u.tcp_ip4_spec.tos)
+ return 1;
+ if (a->m_u.tcp_ip4_spec.tos != b->m_u.tcp_ip4_spec.tos)
+ return 1;
+
+ return 0;
+}
+
+/* Searches the existing flow_specs for flow and return NULL if none found
+ * or the address of the container in the linked list in case of success*/
+static struct ethtool_rx_ntuple_flow_spec_container *gfar_search_table_entry(
+ struct ethtool_rx_ntuple_flow_spec *flow,
+ struct ethtool_rx_ntuple_list *list)
+{
+ struct ethtool_rx_ntuple_flow_spec_container *loop;
+ list_for_each_entry(loop, &list->list, list) {
+ if (gfar_compare_flow_spec(flow, &loop->fs) == 0)
+ return loop;
+ }
+ return NULL;
+}
+
+static int gfar_del_table_entry(
+ struct ethtool_rx_ntuple_flow_spec_container *cont,
+ struct ethtool_rx_ntuple_list *list)
+{
+ list_del(&cont->list);
+ kfree(cont);
+ list->count--;
+ return 0;
+}
+
+static int gfar_process_filer_changes(struct ethtool_rx_ntuple_flow_spec *flow,
+ struct gfar_private *priv)
+{
+ struct ethtool_rx_ntuple_flow_spec_container *j;
+ struct filer_table *tab;
+ s32 i = 0;
+ s32 ret = 0;
+
+ /*So index is set to zero, too!*/
+ tab = kzalloc(sizeof(struct filer_table), GFP_KERNEL);
+ if (tab == NULL) {
+ printk(KERN_WARNING "Can not get memory!\n");
+ return -ENOMEM;
+ }
+
+ j = gfar_search_table_entry(flow, &priv->ntuple_list);
+
+ if (flow->action == ETHTOOL_RXNTUPLE_ACTION_CLEAR) {
+ if (j != NULL)
+ gfar_del_table_entry(j, &priv->ntuple_list);
+ else {
+ printk(KERN_WARNING
+ "Deleting this rule is not possible,"
+ " because it does not exist!\n");
+ return -1;
+ }
+ } else if (j != NULL) {
+ printk(KERN_WARNING "Adding this rule is not possible,"
+ " because it already exists!\n");
+ return -1;
+ }
+
+ /*Now convert the existing filer data from flow_spec into
+ * filer tables binary format*/
+ list_for_each_entry(j, &priv->ntuple_list.list, list) {
+ ret = gfar_convert_to_filer(&j->fs, tab);
+ if (ret == -ESWFULL) {
+ printk(KERN_WARNING
+ "Adding this rule is not possible,"
+ " because there is not space left!\n");
+ goto end;
+ }
+ }
+
+ /*Here add the new one*/
+ if (flow->action != ETHTOOL_RXNTUPLE_ACTION_CLEAR) {
+ ret = gfar_convert_to_filer(flow, tab);
+ if (ret == -ESWFULL) {
+ printk(KERN_WARNING
+ "Adding this rule is not possible,"
+ " because there is not space left!\n");
+ goto end;
+ }
+ if (ret == -1) {
+ printk(KERN_WARNING
+ "Adding this rule is not possible,"
+ " because this flow-type is not supported"
+ " by hardware!\n");
+ goto end;
+ }
+ }
+
+ i = tab->index;
+
+ /*Optimizations to save entries*/
+ gfar_cluster_filer(tab);
+ gfar_optimize_filer_masks(tab);
+
+ printk(KERN_DEBUG "\tSummary:\n"
+ "\tData on hardware: %d\n"
+ "\tCompression rate: %d%%\n", tab->index, 100 - (100 * tab->index) / i);
+
+ /*Write everything to hardware*/
+ ret = gfar_write_filer_table(priv, tab);
+ if (ret == -EHWFULL) {
+ printk(KERN_WARNING
+ "Adding this rule is not possible,"
+ " because there is not space left!\n");
+ goto end;
+ }
+
+ /*Only if all worked fine, add the flow*/
+ if (flow->action != ETHTOOL_RXNTUPLE_ACTION_CLEAR)
+ gfar_add_table_entry(flow, &priv->ntuple_list);
+
+end: kfree(tab);
+ return ret;
+}
+
+static int gfar_set_rx_ntuple(struct net_device *dev,
+ struct ethtool_rx_ntuple *cmd)
+{ struct gfar_private *priv = netdev_priv(dev);
+
+ /*Only values between -2 and num_rx_queues - 1 allowed*/
+ if ((cmd->fs.action >= (signed int)priv->num_rx_queues) ||
+ (cmd->fs.action < ETHTOOL_RXNTUPLE_ACTION_CLEAR))
+ return -EINVAL;
+
+ /* Only one process per device in this region, because the linked list
+ * ntuple_list and the hardware are critical resources*/
+ mutex_lock(&priv->rx_queue_access);
+
+ if (list_empty(&priv->ntuple_list.list))
+ if (gfar_check_filer_hardware(priv) != 0)
+ return -1;
+
+ gfar_process_filer_changes(&cmd->fs, priv);
+
+ mutex_unlock(&priv->rx_queue_access);
+
+ print_hw(priv);
+
+ return 0;
+}
+
+
const struct ethtool_ops gfar_ethtool_ops = {
.get_settings = gfar_gsettings,
.set_settings = gfar_ssettings,
@@ -808,4 +1776,5 @@ const struct ethtool_ops gfar_ethtool_ops = {
.set_wol = gfar_set_wol,
#endif
.set_rxnfc = gfar_set_nfc,
+ .set_rx_ntuple = gfar_set_rx_ntuple
};
^ permalink raw reply related
* Re: [PATCH] gianfar:localized filer table
From: David Miller @ 2011-06-09 7:40 UTC (permalink / raw)
To: s.poehn; +Cc: b06378, netdev, sebastian.poehn
In-Reply-To: <23514.80.254.147.148.1307604607.squirrel@webmail.hs-esslingen.de>
If this is meant to be a real patch submission, don't do it as
a reply to another discussion unless you clearly provide
and delineate a proper full commit log message.
^ permalink raw reply
* Re: [net-next 00/12][pull request] Intel Wired LAN Driver Update
From: Jeff Kirsher @ 2011-06-09 7:46 UTC (permalink / raw)
To: David Miller; +Cc: netdev@vger.kernel.org, gospo@redhat.com
In-Reply-To: <20110609.000823.2202058578852433528.davem@davemloft.net>
[-- Attachment #1: Type: text/plain, Size: 1569 bytes --]
On Thu, 2011-06-09 at 00:08 -0700, David Miller wrote:
> From: David Miller <davem@davemloft.net>
> Date: Thu, 09 Jun 2011 00:05:05 -0700 (PDT)
>
> > From: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
> > Date: Wed, 8 Jun 2011 15:56:28 -0700
> >
> >> The following series is a (smaller) subset of the previous 40 patch
> >> submission. This subset only contains updates to e1000e, igb, igbvf
> >> and rtnetlink.
> >>
> >> e1000e: several cleanups and fixes, as well as bump the version
> >> igb/igbvf/ixgbevf: bump driver version
> >> rtnetlink: Compute and store minimum ifinfo dump size
> >>
> >> The following are changes since commit ffbc03bc75b39c7bd412e7cc6d2185c11b0ffedd:
> >> net: add needed interrupt.h
> >> and are available in the git repository at:
> >> master.kernel.org:/pub/scm/linux/kernel/git/jkirsher/net-next-2.6 master
> >
> > Pulled, thanks Jeff.
>
> Actually, I have to un-pull. Please build test your changes:
>
> drivers/infiniband/core/netlink.c: In function ‘ibnl_rcv_msg’:
> drivers/infiniband/core/netlink.c:151:9: error: too few arguments to function ‘netlink_dump_start’
> include/linux/netlink.h:260:12: note: declared here
> make[3]: *** [drivers/infiniband/core/netlink.o] Error 1
> make[2]: *** [drivers/infiniband/core] Error 2
> make[1]: *** [drivers/infiniband] Error 2
> make[1]: *** Waiting for unfinished jobs....
I built tested after every patch and did not have any issues. I see
Greg made some changes to include/linux/netlink.h which may be the
result of this error.
[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]
^ permalink raw reply
* Re: [net-next 00/12][pull request] Intel Wired LAN Driver Update
From: David Miller @ 2011-06-09 7:50 UTC (permalink / raw)
To: jeffrey.t.kirsher; +Cc: netdev, gospo
In-Reply-To: <1307605611.32044.16.camel@jtkirshe-mobl>
From: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
Date: Thu, 09 Jun 2011 00:46:50 -0700
>> Actually, I have to un-pull. Please build test your changes:
>>
>> drivers/infiniband/core/netlink.c: In function ‘ibnl_rcv_msg’:
>> drivers/infiniband/core/netlink.c:151:9: error: too few arguments to function ‘netlink_dump_start’
>> include/linux/netlink.h:260:12: note: declared here
>> make[3]: *** [drivers/infiniband/core/netlink.o] Error 1
>> make[2]: *** [drivers/infiniband/core] Error 2
>> make[1]: *** [drivers/infiniband] Error 2
>> make[1]: *** Waiting for unfinished jobs....
>
> I built tested after every patch and did not have any issues. I see
> Greg made some changes to include/linux/netlink.h which may be the
> result of this error.
Well of course, the arguments to netlink_dump_start() changed and therefore
every call site needs to be updated.
Unless you don't have infiniband enabled in your builds, you should be
hitting this build failure too.
^ permalink raw reply
* Re: [net-next 00/12][pull request] Intel Wired LAN Driver Update
From: Jeff Kirsher @ 2011-06-09 7:53 UTC (permalink / raw)
To: David Miller; +Cc: netdev@vger.kernel.org, gospo@redhat.com
In-Reply-To: <20110609.005051.1530473933808285841.davem@davemloft.net>
[-- Attachment #1: Type: text/plain, Size: 1293 bytes --]
On Thu, 2011-06-09 at 00:50 -0700, David Miller wrote:
> From: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
> Date: Thu, 09 Jun 2011 00:46:50 -0700
>
> >> Actually, I have to un-pull. Please build test your changes:
> >>
> >> drivers/infiniband/core/netlink.c: In function ‘ibnl_rcv_msg’:
> >> drivers/infiniband/core/netlink.c:151:9: error: too few arguments to function ‘netlink_dump_start’
> >> include/linux/netlink.h:260:12: note: declared here
> >> make[3]: *** [drivers/infiniband/core/netlink.o] Error 1
> >> make[2]: *** [drivers/infiniband/core] Error 2
> >> make[1]: *** [drivers/infiniband] Error 2
> >> make[1]: *** Waiting for unfinished jobs....
> >
> > I built tested after every patch and did not have any issues. I see
> > Greg made some changes to include/linux/netlink.h which may be the
> > result of this error.
>
> Well of course, the arguments to netlink_dump_start() changed and therefore
> every call site needs to be updated.
>
> Unless you don't have infiniband enabled in your builds, you should be
> hitting this build failure too.
Just did, I saw that I did not have enabled and I got the same issue. I
am looking into now, and will check to make sure there were no other
netlink_dump_start() references missed as well.
[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]
^ permalink raw reply
* [PATCH]ethtool.h: fix typos
From: Yegor Yefremov @ 2011-06-09 7:39 UTC (permalink / raw)
To: netdev; +Cc: bhutchings
Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
---
include/linux/ethtool.h | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
Index: b/include/linux/ethtool.h
===================================================================
--- a/include/linux/ethtool.h
+++ b/include/linux/ethtool.h
@@ -268,7 +268,7 @@
__u32 cmd; /* ETHTOOL_{G,S}PAUSEPARAM */
/* If the link is being auto-negotiated (via ethtool_cmd.autoneg
- * being true) the user may set 'autonet' here non-zero to have the
+ * being true) the user may set 'autoneg' here non-zero to have the
* pause parameters be auto-negotiated too. In such a case, the
* {rx,tx}_pause values below determine what capabilities are
* advertised.
@@ -811,7 +811,7 @@
* @get_tx_csum: Deprecated as redundant. Report whether transmit checksums
* are turned on or off.
* @set_tx_csum: Deprecated in favour of generic netdev features. Turn
- * transmit checksums on or off. Returns a egative error code or zero.
+ * transmit checksums on or off. Returns a negative error code or zero.
* @get_sg: Deprecated as redundant. Report whether scatter-gather is
* enabled.
* @set_sg: Deprecated in favour of generic netdev features. Turn
@@ -1087,7 +1087,7 @@
/* The following are all involved in forcing a particular link
* mode for the device for setting things. When getting the
* devices settings, these indicate the current mode and whether
- * it was foced up into this mode or autonegotiated.
+ * it was forced up into this mode or autonegotiated.
*/
/* The forced speed, 10Mb, 100Mb, gigabit, 2.5Gb, 10GbE. */
^ permalink raw reply
* Re: [net-next 00/12][pull request] Intel Wired LAN Driver Update
From: David Miller @ 2011-06-09 7:58 UTC (permalink / raw)
To: jeffrey.t.kirsher; +Cc: netdev, gospo
In-Reply-To: <1307605995.32044.18.camel@jtkirshe-mobl>
From: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
Date: Thu, 09 Jun 2011 00:53:14 -0700
> Just did, I saw that I did not have enabled and I got the same issue. I
> am looking into now, and will check to make sure there were no other
> netlink_dump_start() references missed as well.
Ummm, you're not doing "allmodconfig" builds for validation?
I guarentee you have much more powerful computers than I do and I
do allmodconfig/allyesconfig builds across the board during my work.
And if you're touching global code, you should too :-)
^ permalink raw reply
* [PATCH 00/14] Swap-over-NBD without deadlocking v4r3
From: Mel Gorman @ 2011-06-09 8:02 UTC (permalink / raw)
To: Andrew Morton
Cc: Linux-MM, Linux-Netdev, LKML, David Miller, Neil Brown,
Peter Zijlstra, Mel Gorman
Changelog since V3
o Propogate pfmemalloc from packet fragment pages to skb (Neil)
o Rebase to 3.0-rc2
Changelog since V2
o Document that __GFP_NOMEMALLOC overrides __GFP_MEMALLOC (Neil)
o Use wait_event_interruptible (Neil)
o Use !! when casting to bool to avoid any possibilitity of type
truncation (Neil)
o Nicer logic when using skb_pfmemalloc_protocol (Neil)
Changelog since V1
o Rebase on top of mmotm
o Use atomic_t for memalloc_socks (David Miller)
o Remove use of sk_memalloc_socks in vmscan (Neil Brown)
o Check throttle within prepare_to_wait (Neil Brown)
o Add statistics on throttling instead of printk
Swapping over NBD is something that is technically possible but not
often advised. While there are number of guides on the internet
on how to configure it and nbd-client supports a -swap switch to
"prevent deadlocks", the fact of the matter is a machine using NBD
for swap can be locked up within minutes if swap is used intensively.
The problem is that network block devices do not use mempools like
normal block devices do. As the host cannot control where they receive
packets from, they cannot reliably work out in advance how much memory
they might need.
Some years ago, Peter Ziljstra developed a series of patches that
supported swap over an NFS that some distributions are carrying in
their kernels. This patch series borrows very heavily from Peter's work
to support swapping over NBD (the relatively straight-forward case)
and uses throttling instead of dynamically resized memory reserves
so the series is not too unwieldy for review.
Patch 1 serialises access to min_free_kbytes. It's not strictly needed
by this series but as the series cares about watermarks in
general, it's a harmless fix. It could be merged independently.
Patch 2 adds knowledge of the PFMEMALLOC reserves to SLAB and SLUB to
preserve access to pages allocated under low memory situations
to callers that are freeying memory.
Patch 3 introduces __GFP_MEMALLOC to allow access to the PFMEMALLOC
reserves without setting PFMEMALLOC.
Patch 4 opens the possibility for softirqs to use PFMEMALLOC reserves
for later use by network packet processing.
Patch 5 ignores memory policies when ALLOC_NO_WATERMARKS is set.
Patches 6-10 allows network processing to use PFMEMALLOC reserves when
the socket has been marked as being used by the VM to clean
pages. If packets are received and stored in pages that were
allocated under low-memory situations and are unrelated to
the VM, the packets are dropped.
Patch 11 is a micro-optimisation to avoid a function call in the
common case.
Patch 12 tags NBD sockets as being SOCK_MEMALLOC so they can use
PFMEMALLOC if necessary.
Patch 13 notes that it is still possible for the PFMEMALLOC reserve
to be depleted. To prevent this, direct reclaimers get
throttled on a waitqueue if 50% of the PFMEMALLOC reserves are
depleted. It is expected that kswapd and the direct reclaimers
already running will clean enough pages for the low watermark
to be reached and the throttled processes are woken up.
Patch 14 adds a statistic to track how often processes get throttled
Some basic performance testing was run using kernel builds, netperf
on loopback for UDP and TCP, hackbench (pipes and sockets), iozone
and sysbench. Each of them were expected to use the sl*b allocators
reasonably heavily but there did not appear to be significant
performance variances. Here is the results from netperf using
slab as an example
NETPERF UDP
64 237.47 ( 0.00%) 237.34 (-0.05%)
128 472.69 ( 0.00%) 465.96 (-1.44%)
256 926.82 ( 0.00%) 948.40 ( 2.28%)
1024 3260.08 ( 0.00%) 3266.50 ( 0.20%)
2048 5535.11 ( 0.00%) 5453.55 (-1.50%)
3312 7496.60 ( 0.00%)* 7574.44 ( 1.03%)
1.12% 1.00%
4096 8266.35 ( 0.00%)* 8240.06 (-0.32%)*
1.18% 1.49%
8192 11026.01 ( 0.00%) 11010.44 (-0.14%)
16384 14653.98 ( 0.00%) 14666.97 ( 0.09%)
MMTests Statistics: duration
User/Sys Time Running Test (seconds) 2156.64 1873.27
Total Elapsed Time (seconds) 2570.09 2234.10
NETPERF TCP
netperf-tcp tcp-swapnbd
vanilla-slab v4r3-slab
64 1250.76 ( 0.00%) 1256.52 ( 0.46%)
128 2290.70 ( 0.00%) 2336.43 ( 1.96%)
256 3668.42 ( 0.00%) 3751.17 ( 2.21%)
1024 7214.33 ( 0.00%) 7237.23 ( 0.32%)
2048 8230.01 ( 0.00%) 8280.02 ( 0.60%)
3312 8634.95 ( 0.00%) 8758.62 ( 1.41%)
4096 8851.18 ( 0.00%) 9045.88 ( 2.15%)
8192 10067.59 ( 0.00%) 10263.30 ( 1.91%)
16384 11523.26 ( 0.00%) 11654.78 ( 1.13%)
MMTests Statistics: duration
User/Sys Time Running Test (seconds) 1450.23 1389.8
Total Elapsed Time (seconds) 1450.41 1390.35
Here is the equivalent test for SLUB
netperf-udp udp-swapnbd
vanilla-slub v4r3-slub
64 235.33 ( 0.00%) 237.80 ( 1.04%)
128 465.92 ( 0.00%) 469.98 ( 0.86%)
256 907.16 ( 0.00%) 907.58 ( 0.05%)
1024 3240.25 ( 0.00%) 3255.56 ( 0.47%)
2048 5564.87 ( 0.00%) 5446.46 (-2.17%)
3312 7427.65 ( 0.00%)* 7650.00 ( 2.91%)
1.33% 1.00%
4096 8004.51 ( 0.00%)* 8132.79 ( 1.58%)*
1.05% 1.21%
8192 11079.60 ( 0.00%) 10927.09 (-1.40%)
16384 14737.38 ( 0.00%) 15019.50 ( 1.88%)
MMTests Statistics: duration
User/Sys Time Running Test (seconds) 2056.21 2160.38
Total Elapsed Time (seconds) 2426.09 2498.16
NETPERF TCP
netperf-tcp tcp-swapnbd
vanilla-slub v4r3-slub
64 1251.64 ( 0.00%) 1262.89 ( 0.89%)
128 2289.88 ( 0.00%) 2332.94 ( 1.85%)
256 3654.34 ( 0.00%) 3736.48 ( 2.20%)
1024 7192.47 ( 0.00%) 7286.96 ( 1.30%)
2048 8243.55 ( 0.00%) 8291.50 ( 0.58%)
3312 8664.16 ( 0.00%) 8799.88 ( 1.54%)
4096 8869.13 ( 0.00%) 9018.12 ( 1.65%)
8192 10009.53 ( 0.00%) 10214.26 ( 2.00%)
16384 11470.78 ( 0.00%) 11685.20 ( 1.83%)
MMTests Statistics: duration
User/Sys Time Running Test (seconds) 1368.28 1511.81
Total Elapsed Time (seconds) 1370.33 1510.42
Time to completion varied a lot but this can happen with netperf as
it tries to find results within a sufficiently high confidence. There
were some small gains and losses but they are close to the variances
seen between kernel releases.
For testing swap-over-NBD, a machine was booted with 2G of RAM with a
swapfile backed by NBD. 8*NUM_CPU processes were started that create
anonymous memory mappings and read them linearly in a loop. The total
size of the mappings were 4*PHYSICAL_MEMORY to use swap heavily under
memory pressure. Without the patches, the machine locks up within
minutes and runs to completion with them applied.
drivers/block/nbd.c | 7 +-
include/linux/gfp.h | 13 ++-
include/linux/mm_types.h | 8 ++
include/linux/mmzone.h | 1 +
include/linux/sched.h | 7 +
include/linux/skbuff.h | 21 +++-
include/linux/slub_def.h | 1 +
include/linux/vm_event_item.h | 1 +
include/net/sock.h | 19 +++
include/trace/events/gfpflags.h | 1 +
kernel/softirq.c | 3 +
mm/page_alloc.c | 57 +++++++--
mm/slab.c | 240 +++++++++++++++++++++++++++++++++------
mm/slub.c | 35 +++++-
mm/vmscan.c | 55 +++++++++
mm/vmstat.c | 1 +
net/core/dev.c | 48 +++++++-
net/core/filter.c | 8 ++
net/core/skbuff.c | 95 +++++++++++++---
net/core/sock.c | 42 +++++++
net/ipv4/tcp.c | 3 +-
net/ipv4/tcp_output.c | 13 +-
net/ipv6/tcp_ipv6.c | 12 ++-
23 files changed, 604 insertions(+), 87 deletions(-)
--
1.7.3.4
Mel Gorman (14):
mm: Serialize access to min_free_kbytes
mm: sl[au]b: Add knowledge of PFMEMALLOC reserve pages
mm: Introduce __GFP_MEMALLOC to allow access to emergency reserves
mm: allow PF_MEMALLOC from softirq context
mm: Ignore mempolicies when using ALLOC_NO_WATERMARK
net: Introduce sk_allocation() to allow addition of GFP flags
depending on the individual socket
netvm: Allow the use of __GFP_MEMALLOC by specific sockets
netvm: Allow skb allocation to use PFMEMALLOC reserves
netvm: Propagate page->pfmemalloc to skb
netvm: Set PF_MEMALLOC as appropriate during SKB processing
mm: Micro-optimise slab to avoid a function call
nbd: Set SOCK_MEMALLOC for access to PFMEMALLOC reserves
mm: Throttle direct reclaimers if PF_MEMALLOC reserves are low and
swap is backed by network storage
mm: Account for the number of times direct reclaimers get throttled
drivers/block/nbd.c | 7 +-
include/linux/gfp.h | 13 ++-
include/linux/mm_types.h | 8 ++
include/linux/mmzone.h | 1 +
include/linux/sched.h | 7 +
include/linux/skbuff.h | 21 +++-
include/linux/slub_def.h | 1 +
include/linux/vm_event_item.h | 1 +
include/net/sock.h | 19 +++
include/trace/events/gfpflags.h | 1 +
kernel/softirq.c | 3 +
mm/page_alloc.c | 57 +++++++--
mm/slab.c | 240 +++++++++++++++++++++++++++++++++------
mm/slub.c | 33 +++++-
mm/vmscan.c | 55 +++++++++
mm/vmstat.c | 1 +
net/core/dev.c | 48 +++++++-
net/core/filter.c | 8 ++
net/core/skbuff.c | 95 +++++++++++++---
net/core/sock.c | 42 +++++++
net/ipv4/tcp.c | 3 +-
net/ipv4/tcp_output.c | 13 +-
net/ipv6/tcp_ipv6.c | 12 ++-
23 files changed, 602 insertions(+), 87 deletions(-)
--
1.7.3.4
^ permalink raw reply
* [PATCH 01/14] mm: Serialize access to min_free_kbytes
From: Mel Gorman @ 2011-06-09 8:02 UTC (permalink / raw)
To: Andrew Morton
Cc: Linux-MM, Linux-Netdev, LKML, David Miller, Neil Brown,
Peter Zijlstra, Mel Gorman
In-Reply-To: <1307606573-24704-1-git-send-email-mgorman@suse.de>
There is a race between the min_free_kbytes sysctl, memory hotplug
and transparent hugepage support enablement. Memory hotplug uses a
zonelists_mutex to avoid a race when building zonelists. Reuse it to
serialise watermark updates.
[a.p.zijlstra@chello.nl: Older patch fixed the race with spinlock]
Signed-off-by: Mel Gorman <mgorman@suse.de>
---
mm/page_alloc.c | 23 +++++++++++++++--------
1 files changed, 15 insertions(+), 8 deletions(-)
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 4e8985a..a327a72 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -5045,14 +5045,7 @@ static void setup_per_zone_lowmem_reserve(void)
calculate_totalreserve_pages();
}
-/**
- * setup_per_zone_wmarks - called when min_free_kbytes changes
- * or when memory is hot-{added|removed}
- *
- * Ensures that the watermark[min,low,high] values for each zone are set
- * correctly with respect to min_free_kbytes.
- */
-void setup_per_zone_wmarks(void)
+static void __setup_per_zone_wmarks(void)
{
unsigned long pages_min = min_free_kbytes >> (PAGE_SHIFT - 10);
unsigned long lowmem_pages = 0;
@@ -5107,6 +5100,20 @@ void setup_per_zone_wmarks(void)
calculate_totalreserve_pages();
}
+/**
+ * setup_per_zone_wmarks - called when min_free_kbytes changes
+ * or when memory is hot-{added|removed}
+ *
+ * Ensures that the watermark[min,low,high] values for each zone are set
+ * correctly with respect to min_free_kbytes.
+ */
+void setup_per_zone_wmarks(void)
+{
+ mutex_lock(&zonelists_mutex);
+ __setup_per_zone_wmarks();
+ mutex_unlock(&zonelists_mutex);
+}
+
/*
* The inactive anon list should be small enough that the VM never has to
* do too much work, but large enough that each inactive page has a chance
--
1.7.3.4
^ permalink raw reply related
* [PATCH 02/14] mm: sl[au]b: Add knowledge of PFMEMALLOC reserve pages
From: Mel Gorman @ 2011-06-09 8:02 UTC (permalink / raw)
To: Andrew Morton
Cc: Linux-MM, Linux-Netdev, LKML, David Miller, Neil Brown,
Peter Zijlstra, Mel Gorman
In-Reply-To: <1307606573-24704-1-git-send-email-mgorman@suse.de>
Allocations of pages below the min watermark run a risk of the machine
hanging due to lack of memory. To prevent this, only callers who
have PF_MEMALLOC or TIF_MEMDIE set and not processing an interrupt
are allowed to allocate with ALLOC_NO_WATERMARKS. Once they are
allocated to a slab though, nothing prevents other callers consuming
free objects within those slabs. This patch limits access to slab
pages that were alloced from the PFMEMALLOC reserves.
Pages allocated from the reserve are returned with page->pfmemalloc
set and it's up to the caller to determine how the page should be
protected. SLAB restricts access to any page with page->pfmemalloc set
to callers which are known to able to access the PFMEMALLOC reserve. If
one is not available, an attempt is made to allocate a new page rather
than use a reserve. SLUB is a bit more relaxed in that it only records
if the current per-CPU page was allocated from PFMEMALLOC reserve and
uses another partial slab if the caller does not have the necessary
GFP or process flags. This was found to be sufficient in tests to
avoid hangs due to SLUB generally maintaining smaller lists than SLAB.
In low-memory conditions it does mean that !PFMEMALLOC allocators
can fail a slab allocation even though free objects are available
because they are being preserved for callers that are freeing pages.
[a.p.zijlstra@chello.nl: Original implementation]
Signed-off-by: Mel Gorman <mgorman@suse.de>
---
include/linux/mm_types.h | 8 ++
include/linux/slub_def.h | 1 +
mm/internal.h | 3 +
mm/page_alloc.c | 27 +++++-
mm/slab.c | 216 +++++++++++++++++++++++++++++++++++++++-------
mm/slub.c | 33 ++++++--
6 files changed, 244 insertions(+), 44 deletions(-)
diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index 027935c..2986426 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -71,6 +71,14 @@ struct page {
union {
pgoff_t index; /* Our offset within mapping. */
void *freelist; /* SLUB: freelist req. slab lock */
+ bool pfmemalloc; /* If set by the page allocator,
+ * ALLOC_PFMEMALLOC was set and the
+ * low watermark was not met implying
+ * that the system is under some
+ * pressure. The caller should try
+ * ensure this page is only used to
+ * free other pages.
+ */
};
struct list_head lru; /* Pageout list, eg. active_list
* protected by zone->lru_lock !
diff --git a/include/linux/slub_def.h b/include/linux/slub_def.h
index c8668d1..ff7eb16 100644
--- a/include/linux/slub_def.h
+++ b/include/linux/slub_def.h
@@ -40,6 +40,7 @@ struct kmem_cache_cpu {
unsigned long tid; /* Globally unique transaction id */
struct page *page; /* The slab from which we are allocating */
int node; /* The node of the page (or -1 for debug) */
+ bool pfmemalloc; /* Slab page had pfmemalloc set */
#ifdef CONFIG_SLUB_STATS
unsigned stat[NR_SLUB_STAT_ITEMS];
#endif
diff --git a/mm/internal.h b/mm/internal.h
index d071d380..a520f3b 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -193,6 +193,9 @@ static inline struct page *mem_map_next(struct page *iter,
#define __paginginit __init
#endif
+/* Returns true if the gfp_mask allows use of ALLOC_NO_WATERMARK */
+bool gfp_pfmemalloc_allowed(gfp_t gfp_mask);
+
/* Memory initialisation debug and verification */
enum mminit_level {
MMINIT_WARNING,
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index a327a72..7769a3d 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -656,6 +656,7 @@ static bool free_pages_prepare(struct page *page, unsigned int order)
trace_mm_page_free_direct(page, order);
kmemcheck_free_shadow(page, order);
+ page->pfmemalloc = false;
if (PageAnon(page))
page->mapping = NULL;
for (i = 0; i < (1 << order); i++)
@@ -1174,6 +1175,7 @@ void free_hot_cold_page(struct page *page, int cold)
migratetype = get_pageblock_migratetype(page);
set_page_private(page, migratetype);
+ page->pfmemalloc = false;
local_irq_save(flags);
if (unlikely(wasMlocked))
free_page_mlock(page);
@@ -1367,6 +1369,7 @@ failed:
#define ALLOC_HARDER 0x10 /* try to alloc harder */
#define ALLOC_HIGH 0x20 /* __GFP_HIGH set */
#define ALLOC_CPUSET 0x40 /* check for correct cpuset */
+#define ALLOC_PFMEMALLOC 0x80 /* Caller has PF_MEMALLOC set */
#ifdef CONFIG_FAIL_PAGE_ALLOC
@@ -2041,16 +2044,22 @@ gfp_to_alloc_flags(gfp_t gfp_mask)
} else if (unlikely(rt_task(current)) && !in_interrupt())
alloc_flags |= ALLOC_HARDER;
- if (likely(!(gfp_mask & __GFP_NOMEMALLOC))) {
- if (!in_interrupt() &&
- ((current->flags & PF_MEMALLOC) ||
- unlikely(test_thread_flag(TIF_MEMDIE))))
+ if ((current->flags & PF_MEMALLOC) ||
+ unlikely(test_thread_flag(TIF_MEMDIE))) {
+ alloc_flags |= ALLOC_PFMEMALLOC;
+
+ if (likely(!(gfp_mask & __GFP_NOMEMALLOC)) && !in_interrupt())
alloc_flags |= ALLOC_NO_WATERMARKS;
}
return alloc_flags;
}
+bool gfp_pfmemalloc_allowed(gfp_t gfp_mask)
+{
+ return !!(gfp_to_alloc_flags(gfp_mask) & ALLOC_PFMEMALLOC);
+}
+
static inline struct page *
__alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
struct zonelist *zonelist, enum zone_type high_zoneidx,
@@ -2223,8 +2232,16 @@ nopage:
got_pg:
if (kmemcheck_enabled)
kmemcheck_pagealloc_alloc(page, order, gfp_mask);
- return page;
+ /*
+ * page->pfmemalloc is set when the caller had PFMEMALLOC set or is
+ * been OOM killed. The expectation is that the caller is taking
+ * steps that will free more memory. The caller should avoid the
+ * page being used for !PFMEMALLOC purposes.
+ */
+ page->pfmemalloc = !!(alloc_flags & ALLOC_PFMEMALLOC);
+
+ return page;
}
/*
diff --git a/mm/slab.c b/mm/slab.c
index bcfa498..3ab2db9 100644
--- a/mm/slab.c
+++ b/mm/slab.c
@@ -121,6 +121,8 @@
#include <asm/tlbflush.h>
#include <asm/page.h>
+#include "internal.h"
+
/*
* DEBUG - 1 for kmem_cache_create() to honour; SLAB_RED_ZONE & SLAB_POISON.
* 0 for faster, smaller code (especially in the critical paths).
@@ -227,6 +229,7 @@ struct slab {
unsigned int inuse; /* num of objs active in slab */
kmem_bufctl_t free;
unsigned short nodeid;
+ bool pfmemalloc; /* Slab had pfmemalloc set */
};
struct slab_rcu __slab_cover_slab_rcu;
};
@@ -248,15 +251,37 @@ struct array_cache {
unsigned int avail;
unsigned int limit;
unsigned int batchcount;
- unsigned int touched;
+ bool touched;
+ bool pfmemalloc;
spinlock_t lock;
void *entry[]; /*
* Must have this definition in here for the proper
* alignment of array_cache. Also simplifies accessing
* the entries.
+ *
+ * Entries should not be directly dereferenced as
+ * entries belonging to slabs marked pfmemalloc will
+ * have the lower bits set SLAB_OBJ_PFMEMALLOC
*/
};
+#define SLAB_OBJ_PFMEMALLOC 1
+static inline bool is_obj_pfmemalloc(void *objp)
+{
+ return (unsigned long)objp & SLAB_OBJ_PFMEMALLOC;
+}
+
+static inline void set_obj_pfmemalloc(void **objp)
+{
+ *objp = (void *)((unsigned long)*objp | SLAB_OBJ_PFMEMALLOC);
+ return;
+}
+
+static inline void clear_obj_pfmemalloc(void **objp)
+{
+ *objp = (void *)((unsigned long)*objp & ~SLAB_OBJ_PFMEMALLOC);
+}
+
/*
* bootstrap: The caches do not work without cpuarrays anymore, but the
* cpuarrays are allocated from the generic caches...
@@ -889,12 +914,100 @@ static struct array_cache *alloc_arraycache(int node, int entries,
nc->avail = 0;
nc->limit = entries;
nc->batchcount = batchcount;
- nc->touched = 0;
+ nc->touched = false;
spin_lock_init(&nc->lock);
}
return nc;
}
+/* Clears ac->pfmemalloc if no slabs have pfmalloc set */
+static void check_ac_pfmemalloc(struct kmem_cache *cachep,
+ struct array_cache *ac)
+{
+ struct kmem_list3 *l3 = cachep->nodelists[numa_mem_id()];
+ struct slab *slabp;
+
+ if (!ac->pfmemalloc)
+ return;
+
+ list_for_each_entry(slabp, &l3->slabs_full, list)
+ if (slabp->pfmemalloc)
+ return;
+
+ list_for_each_entry(slabp, &l3->slabs_partial, list)
+ if (slabp->pfmemalloc)
+ return;
+
+ list_for_each_entry(slabp, &l3->slabs_free, list)
+ if (slabp->pfmemalloc)
+ return;
+
+ ac->pfmemalloc = false;
+}
+
+static void *ac_get_obj(struct kmem_cache *cachep, struct array_cache *ac,
+ gfp_t flags, bool force_refill)
+{
+ int i;
+ void *objp = ac->entry[--ac->avail];
+
+ /* Ensure the caller is allowed to use objects from PFMEMALLOC slab */
+ if (unlikely(is_obj_pfmemalloc(objp))) {
+ struct kmem_list3 *l3;
+
+ if (gfp_pfmemalloc_allowed(flags)) {
+ clear_obj_pfmemalloc(&objp);
+ return objp;
+ }
+
+ /* The caller cannot use PFMEMALLOC objects, find another one */
+ for (i = 1; i < ac->avail; i++) {
+ /* If a !PFMEMALLOC object is found, swap them */
+ if (!is_obj_pfmemalloc(ac->entry[i])) {
+ objp = ac->entry[i];
+ ac->entry[i] = ac->entry[ac->avail];
+ ac->entry[ac->avail] = objp;
+ return objp;
+ }
+ }
+
+ /*
+ * If there are empty slabs on the slabs_free list and we are
+ * being forced to refill the cache, mark this one !pfmemalloc.
+ */
+ l3 = cachep->nodelists[numa_mem_id()];
+ if (!list_empty(&l3->slabs_free) && force_refill) {
+ struct slab *slabp = virt_to_slab(objp);
+ slabp->pfmemalloc = false;
+ clear_obj_pfmemalloc(&objp);
+ check_ac_pfmemalloc(cachep, ac);
+ return objp;
+ }
+
+ /* No !PFMEMALLOC objects available */
+ ac->avail++;
+ objp = NULL;
+ }
+
+ return objp;
+}
+
+static void ac_put_obj(struct kmem_cache *cachep, struct array_cache *ac,
+ void *objp)
+{
+ struct slab *slabp;
+
+ /* If there are pfmemalloc slabs, check if the object is part of one */
+ if (unlikely(ac->pfmemalloc)) {
+ slabp = virt_to_slab(objp);
+
+ if (slabp->pfmemalloc)
+ set_obj_pfmemalloc(&objp);
+ }
+
+ ac->entry[ac->avail++] = objp;
+}
+
/*
* Transfer objects in one arraycache to another.
* Locking must be handled by the caller.
@@ -1071,7 +1184,7 @@ static inline int cache_free_alien(struct kmem_cache *cachep, void *objp)
STATS_INC_ACOVERFLOW(cachep);
__drain_alien_cache(cachep, alien, nodeid);
}
- alien->entry[alien->avail++] = objp;
+ ac_put_obj(cachep, alien, objp);
spin_unlock(&alien->lock);
} else {
spin_lock(&(cachep->nodelists[nodeid])->list_lock);
@@ -1678,7 +1791,8 @@ __initcall(cpucache_init);
* did not request dmaable memory, we might get it, but that
* would be relatively rare and ignorable.
*/
-static void *kmem_getpages(struct kmem_cache *cachep, gfp_t flags, int nodeid)
+static void *kmem_getpages(struct kmem_cache *cachep, gfp_t flags, int nodeid,
+ bool *pfmemalloc)
{
struct page *page;
int nr_pages;
@@ -1699,6 +1813,7 @@ static void *kmem_getpages(struct kmem_cache *cachep, gfp_t flags, int nodeid)
page = alloc_pages_exact_node(nodeid, flags | __GFP_NOTRACK, cachep->gfporder);
if (!page)
return NULL;
+ *pfmemalloc = page->pfmemalloc;
nr_pages = (1 << cachep->gfporder);
if (cachep->flags & SLAB_RECLAIM_ACCOUNT)
@@ -2131,7 +2246,7 @@ static int __init_refok setup_cpu_cache(struct kmem_cache *cachep, gfp_t gfp)
cpu_cache_get(cachep)->avail = 0;
cpu_cache_get(cachep)->limit = BOOT_CPUCACHE_ENTRIES;
cpu_cache_get(cachep)->batchcount = 1;
- cpu_cache_get(cachep)->touched = 0;
+ cpu_cache_get(cachep)->touched = false;
cachep->batchcount = 1;
cachep->limit = BOOT_CPUCACHE_ENTRIES;
return 0;
@@ -2678,6 +2793,7 @@ static struct slab *alloc_slabmgmt(struct kmem_cache *cachep, void *objp,
slabp->s_mem = objp + colour_off;
slabp->nodeid = nodeid;
slabp->free = 0;
+ slabp->pfmemalloc = false;
return slabp;
}
@@ -2809,7 +2925,7 @@ static void slab_map_pages(struct kmem_cache *cache, struct slab *slab,
* kmem_cache_alloc() when there are no active objs left in a cache.
*/
static int cache_grow(struct kmem_cache *cachep,
- gfp_t flags, int nodeid, void *objp)
+ gfp_t flags, int nodeid, void *objp, bool pfmemalloc)
{
struct slab *slabp;
size_t offset;
@@ -2853,7 +2969,7 @@ static int cache_grow(struct kmem_cache *cachep,
* 'nodeid'.
*/
if (!objp)
- objp = kmem_getpages(cachep, local_flags, nodeid);
+ objp = kmem_getpages(cachep, local_flags, nodeid, &pfmemalloc);
if (!objp)
goto failed;
@@ -2863,6 +2979,13 @@ static int cache_grow(struct kmem_cache *cachep,
if (!slabp)
goto opps1;
+ /* Record if ALLOC_PFMEMALLOC was set when allocating the slab */
+ if (pfmemalloc) {
+ struct array_cache *ac = cpu_cache_get(cachep);
+ slabp->pfmemalloc = true;
+ ac->pfmemalloc = true;
+ }
+
slab_map_pages(cachep, slabp, objp);
cache_init_objs(cachep, slabp);
@@ -3004,16 +3127,19 @@ bad:
#define check_slabp(x,y) do { } while(0)
#endif
-static void *cache_alloc_refill(struct kmem_cache *cachep, gfp_t flags)
+static void *cache_alloc_refill(struct kmem_cache *cachep, gfp_t flags,
+ bool force_refill)
{
int batchcount;
struct kmem_list3 *l3;
struct array_cache *ac;
int node;
-retry:
check_irq_off();
node = numa_mem_id();
+ if (unlikely(force_refill))
+ goto force_grow;
+retry:
ac = cpu_cache_get(cachep);
batchcount = ac->batchcount;
if (!ac->touched && batchcount > BATCHREFILL_LIMIT) {
@@ -3031,7 +3157,7 @@ retry:
/* See if we can refill from the shared array */
if (l3->shared && transfer_objects(ac, l3->shared, batchcount)) {
- l3->shared->touched = 1;
+ l3->shared->touched = true;
goto alloc_done;
}
@@ -3063,8 +3189,8 @@ retry:
STATS_INC_ACTIVE(cachep);
STATS_SET_HIGH(cachep);
- ac->entry[ac->avail++] = slab_get_obj(cachep, slabp,
- node);
+ ac_put_obj(cachep, ac, slab_get_obj(cachep, slabp,
+ node));
}
check_slabp(cachep, slabp);
@@ -3083,18 +3209,25 @@ alloc_done:
if (unlikely(!ac->avail)) {
int x;
- x = cache_grow(cachep, flags | GFP_THISNODE, node, NULL);
+force_grow:
+ x = cache_grow(cachep, flags | GFP_THISNODE, node, NULL, false);
/* cache_grow can reenable interrupts, then ac could change. */
ac = cpu_cache_get(cachep);
- if (!x && ac->avail == 0) /* no objects in sight? abort */
+
+ /* no objects in sight? abort */
+ if (!x && (ac->avail == 0 || force_refill))
return NULL;
- if (!ac->avail) /* objects refilled by interrupt? */
+ /* objects refilled by interrupt? */
+ if (!ac->avail) {
+ node = numa_node_id();
goto retry;
+ }
}
- ac->touched = 1;
- return ac->entry[--ac->avail];
+ ac->touched = true;
+
+ return ac_get_obj(cachep, ac, flags, force_refill);
}
static inline void cache_alloc_debugcheck_before(struct kmem_cache *cachep,
@@ -3177,23 +3310,35 @@ static inline void *____cache_alloc(struct kmem_cache *cachep, gfp_t flags)
{
void *objp;
struct array_cache *ac;
+ bool force_refill = false;
check_irq_off();
ac = cpu_cache_get(cachep);
if (likely(ac->avail)) {
- STATS_INC_ALLOCHIT(cachep);
- ac->touched = 1;
- objp = ac->entry[--ac->avail];
- } else {
- STATS_INC_ALLOCMISS(cachep);
- objp = cache_alloc_refill(cachep, flags);
+ ac->touched = true;
+ objp = ac_get_obj(cachep, ac, flags, false);
+
/*
- * the 'ac' may be updated by cache_alloc_refill(),
- * and kmemleak_erase() requires its correct value.
+ * Allow for the possibility all avail objects are not allowed
+ * by the current flags
*/
- ac = cpu_cache_get(cachep);
+ if (objp) {
+ STATS_INC_ALLOCHIT(cachep);
+ goto out;
+ }
+ force_refill = true;
}
+
+ STATS_INC_ALLOCMISS(cachep);
+ objp = cache_alloc_refill(cachep, flags, force_refill);
+ /*
+ * the 'ac' may be updated by cache_alloc_refill(),
+ * and kmemleak_erase() requires its correct value.
+ */
+ ac = cpu_cache_get(cachep);
+
+out:
/*
* To avoid a false negative, if an object that is in one of the
* per-CPU caches is leaked, we need to make sure kmemleak doesn't
@@ -3246,6 +3391,7 @@ static void *fallback_alloc(struct kmem_cache *cache, gfp_t flags)
enum zone_type high_zoneidx = gfp_zone(flags);
void *obj = NULL;
int nid;
+ bool pfmemalloc;
if (flags & __GFP_THISNODE)
return NULL;
@@ -3282,7 +3428,8 @@ retry:
if (local_flags & __GFP_WAIT)
local_irq_enable();
kmem_flagcheck(cache, flags);
- obj = kmem_getpages(cache, local_flags, numa_mem_id());
+ obj = kmem_getpages(cache, local_flags, numa_mem_id(),
+ &pfmemalloc);
if (local_flags & __GFP_WAIT)
local_irq_disable();
if (obj) {
@@ -3290,7 +3437,7 @@ retry:
* Insert into the appropriate per node queues
*/
nid = page_to_nid(virt_to_page(obj));
- if (cache_grow(cache, flags, nid, obj)) {
+ if (cache_grow(cache, flags, nid, obj, pfmemalloc)) {
obj = ____cache_alloc_node(cache,
flags | GFP_THISNODE, nid);
if (!obj)
@@ -3362,7 +3509,7 @@ retry:
must_grow:
spin_unlock(&l3->list_lock);
- x = cache_grow(cachep, flags | GFP_THISNODE, nodeid, NULL);
+ x = cache_grow(cachep, flags | GFP_THISNODE, nodeid, NULL, false);
if (x)
goto retry;
@@ -3512,9 +3659,12 @@ static void free_block(struct kmem_cache *cachep, void **objpp, int nr_objects,
struct kmem_list3 *l3;
for (i = 0; i < nr_objects; i++) {
- void *objp = objpp[i];
+ void *objp;
struct slab *slabp;
+ clear_obj_pfmemalloc(&objpp[i]);
+ objp = objpp[i];
+
slabp = virt_to_slab(objp);
l3 = cachep->nodelists[node];
list_del(&slabp->list);
@@ -3626,12 +3776,12 @@ static inline void __cache_free(struct kmem_cache *cachep, void *objp)
if (likely(ac->avail < ac->limit)) {
STATS_INC_FREEHIT(cachep);
- ac->entry[ac->avail++] = objp;
+ ac_put_obj(cachep, ac, objp);
return;
} else {
STATS_INC_FREEMISS(cachep);
cache_flusharray(cachep, ac);
- ac->entry[ac->avail++] = objp;
+ ac_put_obj(cachep, ac, objp);
}
}
@@ -4057,7 +4207,7 @@ static void drain_array(struct kmem_cache *cachep, struct kmem_list3 *l3,
if (!ac || !ac->avail)
return;
if (ac->touched && !force) {
- ac->touched = 0;
+ ac->touched = false;
} else {
spin_lock_irq(&l3->list_lock);
if (ac->avail) {
diff --git a/mm/slub.c b/mm/slub.c
index 7be0223..267538a 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -30,6 +30,8 @@
#include <trace/events/kmem.h>
+#include "internal.h"
+
/*
* Lock order:
* 1. slab_lock(page)
@@ -1242,7 +1244,8 @@ static void setup_object(struct kmem_cache *s, struct page *page,
s->ctor(object);
}
-static struct page *new_slab(struct kmem_cache *s, gfp_t flags, int node)
+static struct page *new_slab(struct kmem_cache *s, gfp_t flags, int node,
+ bool *pfmemalloc)
{
struct page *page;
void *start;
@@ -1257,6 +1260,7 @@ static struct page *new_slab(struct kmem_cache *s, gfp_t flags, int node)
goto out;
inc_slabs_node(s, page_to_nid(page), page->objects);
+ *pfmemalloc = page->pfmemalloc;
page->slab = s;
page->flags |= 1 << PG_slab;
@@ -1772,6 +1776,14 @@ slab_out_of_memory(struct kmem_cache *s, gfp_t gfpflags, int nid)
}
}
+static inline bool pfmemalloc_match(struct kmem_cache_cpu *c, gfp_t gfpflags)
+{
+ if (unlikely(c->pfmemalloc))
+ return gfp_pfmemalloc_allowed(gfpflags);
+
+ return true;
+}
+
/*
* Slow path. The lockless freelist is empty or we need to perform
* debugging duties.
@@ -1796,6 +1808,7 @@ static void *__slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node,
void **object;
struct page *page;
unsigned long flags;
+ bool pfmemalloc = false;
local_irq_save(flags);
#ifdef CONFIG_PREEMPT
@@ -1815,7 +1828,13 @@ static void *__slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node,
goto new_slab;
slab_lock(page);
- if (unlikely(!node_match(c, node)))
+
+ /*
+ * By rights, we should be searching for a slab page that was
+ * PFMEMALLOC but right now, we are losing the pfmemalloc
+ * information when the page leaves the per-cpu allocator
+ */
+ if (unlikely(!pfmemalloc_match(c, gfpflags) || !node_match(c, node)))
goto another_slab;
stat(s, ALLOC_REFILL);
@@ -1853,7 +1872,7 @@ new_slab:
if (gfpflags & __GFP_WAIT)
local_irq_enable();
- page = new_slab(s, gfpflags, node);
+ page = new_slab(s, gfpflags, node, &pfmemalloc);
if (gfpflags & __GFP_WAIT)
local_irq_disable();
@@ -1868,6 +1887,7 @@ new_slab:
__SetPageSlubFrozen(page);
c->node = page_to_nid(page);
c->page = page;
+ c->pfmemalloc = pfmemalloc;
goto load_freelist;
}
if (!(gfpflags & __GFP_NOWARN) && printk_ratelimit())
@@ -1927,8 +1947,8 @@ redo:
barrier();
object = c->freelist;
- if (unlikely(!object || !node_match(c, node)))
-
+ if (unlikely(!object || !node_match(c, node) ||
+ !pfmemalloc_match(c, gfpflags)))
object = __slab_alloc(s, gfpflags, node, addr, c);
else {
@@ -2355,10 +2375,11 @@ static void early_kmem_cache_node_alloc(int node)
struct page *page;
struct kmem_cache_node *n;
unsigned long flags;
+ bool pfmemalloc; /* Ignore this early in boot */
BUG_ON(kmem_cache_node->size < sizeof(struct kmem_cache_node));
- page = new_slab(kmem_cache_node, GFP_NOWAIT, node);
+ page = new_slab(kmem_cache_node, GFP_NOWAIT, node, &pfmemalloc);
BUG_ON(!page);
if (page_to_nid(page) != node) {
--
1.7.3.4
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* [PATCH 03/14] mm: Introduce __GFP_MEMALLOC to allow access to emergency reserves
From: Mel Gorman @ 2011-06-09 8:02 UTC (permalink / raw)
To: Andrew Morton
Cc: Linux-MM, Linux-Netdev, LKML, David Miller, Neil Brown,
Peter Zijlstra, Mel Gorman
In-Reply-To: <1307606573-24704-1-git-send-email-mgorman@suse.de>
__GFP_MEMALLOC will allow the allocation to disregard the watermarks,
much like PF_MEMALLOC. It allows one to pass along the memalloc state in
object related allocation flags as opposed to task related flags, such
as sk->sk_allocation. This removes the need for ALLOC_PFMEMALLOC as
callers using __GFP_MEMALLOC can get the ALLOC_NO_WATERMARK flag which
is now enough to identify allocations related to page reclaim.
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
Signed-off-by: Mel Gorman <mgorman@suse.de>
---
include/linux/gfp.h | 10 ++++++++--
include/linux/mm_types.h | 2 +-
include/trace/events/gfpflags.h | 1 +
mm/page_alloc.c | 14 ++++++--------
mm/slab.c | 2 +-
5 files changed, 17 insertions(+), 12 deletions(-)
diff --git a/include/linux/gfp.h b/include/linux/gfp.h
index cb40892..faefba9 100644
--- a/include/linux/gfp.h
+++ b/include/linux/gfp.h
@@ -23,6 +23,7 @@ struct vm_area_struct;
#define ___GFP_REPEAT 0x400u
#define ___GFP_NOFAIL 0x800u
#define ___GFP_NORETRY 0x1000u
+#define ___GFP_MEMALLOC 0x2000u
#define ___GFP_COMP 0x4000u
#define ___GFP_ZERO 0x8000u
#define ___GFP_NOMEMALLOC 0x10000u
@@ -75,9 +76,14 @@ struct vm_area_struct;
#define __GFP_REPEAT ((__force gfp_t)___GFP_REPEAT) /* See above */
#define __GFP_NOFAIL ((__force gfp_t)___GFP_NOFAIL) /* See above */
#define __GFP_NORETRY ((__force gfp_t)___GFP_NORETRY) /* See above */
+#define __GFP_MEMALLOC ((__force gfp_t)___GFP_MEMALLOC)/* Allow access to emergency reserves */
#define __GFP_COMP ((__force gfp_t)___GFP_COMP) /* Add compound page metadata */
#define __GFP_ZERO ((__force gfp_t)___GFP_ZERO) /* Return zeroed page on success */
-#define __GFP_NOMEMALLOC ((__force gfp_t)___GFP_NOMEMALLOC) /* Don't use emergency reserves */
+#define __GFP_NOMEMALLOC ((__force gfp_t)___GFP_NOMEMALLOC) /* Don't use emergency reserves.
+ * This takes precedence over the
+ * __GFP_MEMALLOC flag if both are
+ * set
+ */
#define __GFP_HARDWALL ((__force gfp_t)___GFP_HARDWALL) /* Enforce hardwall cpuset memory allocs */
#define __GFP_THISNODE ((__force gfp_t)___GFP_THISNODE)/* No fallback, no policies */
#define __GFP_RECLAIMABLE ((__force gfp_t)___GFP_RECLAIMABLE) /* Page is reclaimable */
@@ -127,7 +133,7 @@ struct vm_area_struct;
/* Control page allocator reclaim behavior */
#define GFP_RECLAIM_MASK (__GFP_WAIT|__GFP_HIGH|__GFP_IO|__GFP_FS|\
__GFP_NOWARN|__GFP_REPEAT|__GFP_NOFAIL|\
- __GFP_NORETRY|__GFP_NOMEMALLOC)
+ __GFP_NORETRY|__GFP_MEMALLOC|__GFP_NOMEMALLOC)
/* Control slab gfp mask during early boot */
#define GFP_BOOT_MASK (__GFP_BITS_MASK & ~(__GFP_WAIT|__GFP_IO|__GFP_FS))
diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index 2986426..319a7cd 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -72,7 +72,7 @@ struct page {
pgoff_t index; /* Our offset within mapping. */
void *freelist; /* SLUB: freelist req. slab lock */
bool pfmemalloc; /* If set by the page allocator,
- * ALLOC_PFMEMALLOC was set and the
+ * ALLOC_NO_WATERMARKS was set and the
* low watermark was not met implying
* that the system is under some
* pressure. The caller should try
diff --git a/include/trace/events/gfpflags.h b/include/trace/events/gfpflags.h
index 9fe3a366..d6fd8e5 100644
--- a/include/trace/events/gfpflags.h
+++ b/include/trace/events/gfpflags.h
@@ -30,6 +30,7 @@
{(unsigned long)__GFP_COMP, "GFP_COMP"}, \
{(unsigned long)__GFP_ZERO, "GFP_ZERO"}, \
{(unsigned long)__GFP_NOMEMALLOC, "GFP_NOMEMALLOC"}, \
+ {(unsigned long)__GFP_MEMALLOC, "GFP_MEMALLOC"}, \
{(unsigned long)__GFP_HARDWALL, "GFP_HARDWALL"}, \
{(unsigned long)__GFP_THISNODE, "GFP_THISNODE"}, \
{(unsigned long)__GFP_RECLAIMABLE, "GFP_RECLAIMABLE"}, \
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 7769a3d..27043e7 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -1369,7 +1369,6 @@ failed:
#define ALLOC_HARDER 0x10 /* try to alloc harder */
#define ALLOC_HIGH 0x20 /* __GFP_HIGH set */
#define ALLOC_CPUSET 0x40 /* check for correct cpuset */
-#define ALLOC_PFMEMALLOC 0x80 /* Caller has PF_MEMALLOC set */
#ifdef CONFIG_FAIL_PAGE_ALLOC
@@ -2044,11 +2043,10 @@ gfp_to_alloc_flags(gfp_t gfp_mask)
} else if (unlikely(rt_task(current)) && !in_interrupt())
alloc_flags |= ALLOC_HARDER;
- if ((current->flags & PF_MEMALLOC) ||
- unlikely(test_thread_flag(TIF_MEMDIE))) {
- alloc_flags |= ALLOC_PFMEMALLOC;
-
- if (likely(!(gfp_mask & __GFP_NOMEMALLOC)) && !in_interrupt())
+ if (likely(!(gfp_mask & __GFP_NOMEMALLOC))) {
+ if (gfp_mask & __GFP_MEMALLOC)
+ alloc_flags |= ALLOC_NO_WATERMARKS;
+ else if (likely(!(gfp_mask & __GFP_NOMEMALLOC)) && !in_interrupt())
alloc_flags |= ALLOC_NO_WATERMARKS;
}
@@ -2057,7 +2055,7 @@ gfp_to_alloc_flags(gfp_t gfp_mask)
bool gfp_pfmemalloc_allowed(gfp_t gfp_mask)
{
- return !!(gfp_to_alloc_flags(gfp_mask) & ALLOC_PFMEMALLOC);
+ return !!(gfp_to_alloc_flags(gfp_mask) & ALLOC_NO_WATERMARKS);
}
static inline struct page *
@@ -2239,7 +2237,7 @@ got_pg:
* steps that will free more memory. The caller should avoid the
* page being used for !PFMEMALLOC purposes.
*/
- page->pfmemalloc = !!(alloc_flags & ALLOC_PFMEMALLOC);
+ page->pfmemalloc = !!(alloc_flags & ALLOC_NO_WATERMARKS);
return page;
}
diff --git a/mm/slab.c b/mm/slab.c
index 3ab2db9..708da2f 100644
--- a/mm/slab.c
+++ b/mm/slab.c
@@ -2979,7 +2979,7 @@ static int cache_grow(struct kmem_cache *cachep,
if (!slabp)
goto opps1;
- /* Record if ALLOC_PFMEMALLOC was set when allocating the slab */
+ /* Record if ALLOC_NO_WATERMARKS was set when allocating the slab */
if (pfmemalloc) {
struct array_cache *ac = cpu_cache_get(cachep);
slabp->pfmemalloc = true;
--
1.7.3.4
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* [PATCH 04/14] mm: allow PF_MEMALLOC from softirq context
From: Mel Gorman @ 2011-06-09 8:02 UTC (permalink / raw)
To: Andrew Morton
Cc: Linux-MM, Linux-Netdev, LKML, David Miller, Neil Brown,
Peter Zijlstra, Mel Gorman
In-Reply-To: <1307606573-24704-1-git-send-email-mgorman@suse.de>
This is needed to allow network softirq packet processing to make use
of PF_MEMALLOC.
Currently softirq context cannot use PF_MEMALLOC due to it not being
associated with a task, and therefore not having task flags to fiddle with -
thus the gfp to alloc flag mapping ignores the task flags when in interrupts
(hard or soft) context.
Allowing softirqs to make use of PF_MEMALLOC therefore requires some trickery.
We basically borrow the task flags from whatever process happens to be
preempted by the softirq.
So we modify the gfp to alloc flags mapping to not exclude task flags in
softirq context, and modify the softirq code to save, clear and restore
the PF_MEMALLOC flag.
The save and clear, ensures the preempted task's PF_MEMALLOC flag doesn't
leak into the softirq. The restore ensures a softirq's PF_MEMALLOC flag
cannot leak back into the preempted process.
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
Signed-off-by: Mel Gorman <mgorman@suse.de>
---
include/linux/sched.h | 7 +++++++
kernel/softirq.c | 3 +++
mm/page_alloc.c | 5 ++++-
3 files changed, 14 insertions(+), 1 deletions(-)
diff --git a/include/linux/sched.h b/include/linux/sched.h
index 2a8621c..7f908f9 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -1840,6 +1840,13 @@ static inline void rcu_copy_process(struct task_struct *p)
#endif
+static inline void tsk_restore_flags(struct task_struct *p,
+ unsigned long pflags, unsigned long mask)
+{
+ p->flags &= ~mask;
+ p->flags |= pflags & mask;
+}
+
#ifdef CONFIG_SMP
extern void do_set_cpus_allowed(struct task_struct *p,
const struct cpumask *new_mask);
diff --git a/kernel/softirq.c b/kernel/softirq.c
index 1396017..2817c27 100644
--- a/kernel/softirq.c
+++ b/kernel/softirq.c
@@ -210,6 +210,8 @@ asmlinkage void __do_softirq(void)
__u32 pending;
int max_restart = MAX_SOFTIRQ_RESTART;
int cpu;
+ unsigned long pflags = current->flags;
+ current->flags &= ~PF_MEMALLOC;
pending = local_softirq_pending();
account_system_vtime(current);
@@ -265,6 +267,7 @@ restart:
account_system_vtime(current);
__local_bh_enable(SOFTIRQ_OFFSET);
+ tsk_restore_flags(current, pflags, PF_MEMALLOC);
}
#ifndef __ARCH_HAS_DO_SOFTIRQ
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 27043e7..4e19606 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -2046,7 +2046,10 @@ gfp_to_alloc_flags(gfp_t gfp_mask)
if (likely(!(gfp_mask & __GFP_NOMEMALLOC))) {
if (gfp_mask & __GFP_MEMALLOC)
alloc_flags |= ALLOC_NO_WATERMARKS;
- else if (likely(!(gfp_mask & __GFP_NOMEMALLOC)) && !in_interrupt())
+ else if (!in_irq() && (current->flags & PF_MEMALLOC))
+ alloc_flags |= ALLOC_NO_WATERMARKS;
+ else if (!in_interrupt() &&
+ unlikely(test_thread_flag(TIF_MEMDIE)))
alloc_flags |= ALLOC_NO_WATERMARKS;
}
--
1.7.3.4
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* [PATCH 05/14] mm: Ignore mempolicies when using ALLOC_NO_WATERMARK
From: Mel Gorman @ 2011-06-09 8:02 UTC (permalink / raw)
To: Andrew Morton
Cc: Linux-MM, Linux-Netdev, LKML, David Miller, Neil Brown,
Peter Zijlstra, Mel Gorman
In-Reply-To: <1307606573-24704-1-git-send-email-mgorman@suse.de>
The reserve is proportionally distributed over all !highmem zones in the
system. So we need to allow an emergency allocation access to all zones.
In order to do that we need to break out of any mempolicy boundaries we
might have.
In my opinion that does not break mempolicies as those are user oriented
and not system oriented. That is, system allocations are not guaranteed to
be within mempolicy boundaries. For instance IRQs don't even have a mempolicy.
So breaking out of mempolicy boundaries for 'rare' emergency allocations,
which are always system allocations (as opposed to user) is ok.
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
Signed-off-by: Mel Gorman <mgorman@suse.de>
---
mm/page_alloc.c | 7 +++++++
1 files changed, 7 insertions(+), 0 deletions(-)
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 4e19606..ac779f5 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -2126,6 +2126,13 @@ rebalance:
/* Allocate without watermarks if the context allows */
if (alloc_flags & ALLOC_NO_WATERMARKS) {
+ /*
+ * Ignore mempolicies if ALLOC_NO_WATERMARKS on the grounds
+ * the allocation is high priority and these type of
+ * allocations are system rather than user orientated
+ */
+ zonelist = node_zonelist(numa_node_id(), gfp_mask);
+
page = __alloc_pages_high_priority(gfp_mask, order,
zonelist, high_zoneidx, nodemask,
preferred_zone, migratetype);
--
1.7.3.4
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* [PATCH 06/14] net: Introduce sk_allocation() to allow addition of GFP flags depending on the individual socket
From: Mel Gorman @ 2011-06-09 8:02 UTC (permalink / raw)
To: Andrew Morton
Cc: Linux-MM, Linux-Netdev, LKML, David Miller, Neil Brown,
Peter Zijlstra, Mel Gorman
In-Reply-To: <1307606573-24704-1-git-send-email-mgorman@suse.de>
Introduce sk_allocation(), this function allows to inject sock specific
flags to each sock related allocation. It is only used on allocation
paths that may be required for writing pages back to network storage.
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
Signed-off-by: Mel Gorman <mgorman@suse.de>
---
include/net/sock.h | 5 +++++
net/ipv4/tcp.c | 3 ++-
net/ipv4/tcp_output.c | 13 +++++++------
net/ipv6/tcp_ipv6.c | 12 +++++++++---
4 files changed, 23 insertions(+), 10 deletions(-)
diff --git a/include/net/sock.h b/include/net/sock.h
index f2046e4..e89c38f 100644
--- a/include/net/sock.h
+++ b/include/net/sock.h
@@ -585,6 +585,11 @@ static inline int sock_flag(struct sock *sk, enum sock_flags flag)
return test_bit(flag, &sk->sk_flags);
}
+static inline gfp_t sk_allocation(struct sock *sk, gfp_t gfp_mask)
+{
+ return gfp_mask;
+}
+
static inline void sk_acceptq_removed(struct sock *sk)
{
sk->sk_ack_backlog--;
diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c
index 054a59d..8c1a9d5 100644
--- a/net/ipv4/tcp.c
+++ b/net/ipv4/tcp.c
@@ -698,7 +698,8 @@ struct sk_buff *sk_stream_alloc_skb(struct sock *sk, int size, gfp_t gfp)
/* The TCP header must be at least 32-bit aligned. */
size = ALIGN(size, 4);
- skb = alloc_skb_fclone(size + sk->sk_prot->max_header, gfp);
+ skb = alloc_skb_fclone(size + sk->sk_prot->max_header,
+ sk_allocation(sk, gfp));
if (skb) {
if (sk_wmem_schedule(sk, skb->truesize)) {
/*
diff --git a/net/ipv4/tcp_output.c b/net/ipv4/tcp_output.c
index 882e0b0..87b98f6 100644
--- a/net/ipv4/tcp_output.c
+++ b/net/ipv4/tcp_output.c
@@ -2324,7 +2324,7 @@ void tcp_send_fin(struct sock *sk)
/* Socket is locked, keep trying until memory is available. */
for (;;) {
skb = alloc_skb_fclone(MAX_TCP_HEADER,
- sk->sk_allocation);
+ sk_allocation(sk, GFP_KERNEL));
if (skb)
break;
yield();
@@ -2350,7 +2350,7 @@ void tcp_send_active_reset(struct sock *sk, gfp_t priority)
struct sk_buff *skb;
/* NOTE: No TCP options attached and we never retransmit this. */
- skb = alloc_skb(MAX_TCP_HEADER, priority);
+ skb = alloc_skb(MAX_TCP_HEADER, sk_allocation(sk, priority));
if (!skb) {
NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPABORTFAILED);
return;
@@ -2423,7 +2423,8 @@ struct sk_buff *tcp_make_synack(struct sock *sk, struct dst_entry *dst,
if (cvp != NULL && cvp->s_data_constant && cvp->s_data_desired)
s_data_desired = cvp->s_data_desired;
- skb = sock_wmalloc(sk, MAX_TCP_HEADER + 15 + s_data_desired, 1, GFP_ATOMIC);
+ skb = sock_wmalloc(sk, MAX_TCP_HEADER + 15 + s_data_desired, 1,
+ sk_allocation(sk, GFP_ATOMIC));
if (skb == NULL)
return NULL;
@@ -2719,7 +2720,7 @@ void tcp_send_ack(struct sock *sk)
* tcp_transmit_skb() will set the ownership to this
* sock.
*/
- buff = alloc_skb(MAX_TCP_HEADER, GFP_ATOMIC);
+ buff = alloc_skb(MAX_TCP_HEADER, sk_allocation(sk, GFP_ATOMIC));
if (buff == NULL) {
inet_csk_schedule_ack(sk);
inet_csk(sk)->icsk_ack.ato = TCP_ATO_MIN;
@@ -2734,7 +2735,7 @@ void tcp_send_ack(struct sock *sk)
/* Send it off, this clears delayed acks for us. */
TCP_SKB_CB(buff)->when = tcp_time_stamp;
- tcp_transmit_skb(sk, buff, 0, GFP_ATOMIC);
+ tcp_transmit_skb(sk, buff, 0, sk_allocation(sk, GFP_ATOMIC));
}
/* This routine sends a packet with an out of date sequence
@@ -2754,7 +2755,7 @@ static int tcp_xmit_probe_skb(struct sock *sk, int urgent)
struct sk_buff *skb;
/* We don't queue it, tcp_transmit_skb() sets ownership. */
- skb = alloc_skb(MAX_TCP_HEADER, GFP_ATOMIC);
+ skb = alloc_skb(MAX_TCP_HEADER, sk_allocation(sk, GFP_ATOMIC));
if (skb == NULL)
return -1;
diff --git a/net/ipv6/tcp_ipv6.c b/net/ipv6/tcp_ipv6.c
index d1fd287..62bc424 100644
--- a/net/ipv6/tcp_ipv6.c
+++ b/net/ipv6/tcp_ipv6.c
@@ -597,7 +597,8 @@ static int tcp_v6_md5_do_add(struct sock *sk, const struct in6_addr *peer,
} else {
/* reallocate new list if current one is full. */
if (!tp->md5sig_info) {
- tp->md5sig_info = kzalloc(sizeof(*tp->md5sig_info), GFP_ATOMIC);
+ tp->md5sig_info = kzalloc(sizeof(*tp->md5sig_info),
+ sk_allocation(sk, GFP_ATOMIC));
if (!tp->md5sig_info) {
kfree(newkey);
return -ENOMEM;
@@ -610,7 +611,8 @@ static int tcp_v6_md5_do_add(struct sock *sk, const struct in6_addr *peer,
}
if (tp->md5sig_info->alloced6 == tp->md5sig_info->entries6) {
keys = kmalloc((sizeof (tp->md5sig_info->keys6[0]) *
- (tp->md5sig_info->entries6 + 1)), GFP_ATOMIC);
+ (tp->md5sig_info->entries6 + 1)),
+ sk_allocation(sk, GFP_ATOMIC));
if (!keys) {
tcp_free_md5sig_pool();
@@ -734,7 +736,8 @@ static int tcp_v6_parse_md5_keys (struct sock *sk, char __user *optval,
struct tcp_sock *tp = tcp_sk(sk);
struct tcp_md5sig_info *p;
- p = kzalloc(sizeof(struct tcp_md5sig_info), GFP_KERNEL);
+ p = kzalloc(sizeof(struct tcp_md5sig_info),
+ sk_allocation(sk, GFP_KERNEL));
if (!p)
return -ENOMEM;
@@ -1084,6 +1087,7 @@ static void tcp_v6_send_reset(struct sock *sk, struct sk_buff *skb)
struct tcphdr *th = tcp_hdr(skb);
u32 seq = 0, ack_seq = 0;
struct tcp_md5sig_key *key = NULL;
+ gfp_t gfp_mask = GFP_ATOMIC;
if (th->rst)
return;
@@ -1095,6 +1099,8 @@ static void tcp_v6_send_reset(struct sock *sk, struct sk_buff *skb)
if (sk)
key = tcp_v6_md5_do_lookup(sk, &ipv6_hdr(skb)->daddr);
#endif
+ if (sk)
+ gfp_mask = sk_allocation(sk, gfp_mask);
if (th->ack)
seq = ntohl(th->ack_seq);
--
1.7.3.4
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* [PATCH 07/14] netvm: Allow the use of __GFP_MEMALLOC by specific sockets
From: Mel Gorman @ 2011-06-09 8:02 UTC (permalink / raw)
To: Andrew Morton
Cc: Linux-MM, Linux-Netdev, LKML, David Miller, Neil Brown,
Peter Zijlstra, Mel Gorman
In-Reply-To: <1307606573-24704-1-git-send-email-mgorman@suse.de>
Allow specific sockets to be tagged SOCK_MEMALLOC and use __GFP_MEMALLOC
for their allocations. These sockets will be able to go below watermarks
and allocate from the emergency reserve. Such sockets are to be used
to service the VM (iow. to swap over). They must be handled kernel side,
exposing such a socket to user-space is a bug.
There is a risk that the reserves be depleted so for now, the administrator is
responsible for increasing min_free_kbytes as necessary to prevent deadlock
for their workloads.
[a.p.zijlstra@chello.nl: Original patches]
Signed-off-by: Mel Gorman <mgorman@suse.de>
---
include/net/sock.h | 5 ++++-
net/core/sock.c | 22 ++++++++++++++++++++++
2 files changed, 26 insertions(+), 1 deletions(-)
diff --git a/include/net/sock.h b/include/net/sock.h
index e89c38f..046bc97 100644
--- a/include/net/sock.h
+++ b/include/net/sock.h
@@ -554,6 +554,7 @@ enum sock_flags {
SOCK_RCVTSTAMPNS, /* %SO_TIMESTAMPNS setting */
SOCK_LOCALROUTE, /* route locally only, %SO_DONTROUTE setting */
SOCK_QUEUE_SHRUNK, /* write queue has been shrunk recently */
+ SOCK_MEMALLOC, /* VM depends on this socket for swapping */
SOCK_TIMESTAMPING_TX_HARDWARE, /* %SOF_TIMESTAMPING_TX_HARDWARE */
SOCK_TIMESTAMPING_TX_SOFTWARE, /* %SOF_TIMESTAMPING_TX_SOFTWARE */
SOCK_TIMESTAMPING_RX_HARDWARE, /* %SOF_TIMESTAMPING_RX_HARDWARE */
@@ -587,7 +588,7 @@ static inline int sock_flag(struct sock *sk, enum sock_flags flag)
static inline gfp_t sk_allocation(struct sock *sk, gfp_t gfp_mask)
{
- return gfp_mask;
+ return gfp_mask | (sk->sk_allocation & __GFP_MEMALLOC);
}
static inline void sk_acceptq_removed(struct sock *sk)
@@ -717,6 +718,8 @@ extern int sk_stream_wait_memory(struct sock *sk, long *timeo_p);
extern void sk_stream_wait_close(struct sock *sk, long timeo_p);
extern int sk_stream_error(struct sock *sk, int flags, int err);
extern void sk_stream_kill_queues(struct sock *sk);
+extern void sk_set_memalloc(struct sock *sk);
+extern void sk_clear_memalloc(struct sock *sk);
extern int sk_wait_data(struct sock *sk, long *timeo);
diff --git a/net/core/sock.c b/net/core/sock.c
index 6e81978..c685eda 100644
--- a/net/core/sock.c
+++ b/net/core/sock.c
@@ -219,6 +219,28 @@ __u32 sysctl_rmem_default __read_mostly = SK_RMEM_MAX;
int sysctl_optmem_max __read_mostly = sizeof(unsigned long)*(2*UIO_MAXIOV+512);
EXPORT_SYMBOL(sysctl_optmem_max);
+/**
+ * sk_set_memalloc - sets %SOCK_MEMALLOC
+ * @sk: socket to set it on
+ *
+ * Set %SOCK_MEMALLOC on a socket for access to emergency reserves.
+ * It's the responsibility of the admin to adjust min_free_kbytes
+ * to meet the requirements
+ */
+void sk_set_memalloc(struct sock *sk)
+{
+ sock_set_flag(sk, SOCK_MEMALLOC);
+ sk->sk_allocation |= __GFP_MEMALLOC;
+}
+EXPORT_SYMBOL_GPL(sk_set_memalloc);
+
+void sk_clear_memalloc(struct sock *sk)
+{
+ sock_reset_flag(sk, SOCK_MEMALLOC);
+ sk->sk_allocation &= ~__GFP_MEMALLOC;
+}
+EXPORT_SYMBOL_GPL(sk_clear_memalloc);
+
#if defined(CONFIG_CGROUPS) && !defined(CONFIG_NET_CLS_CGROUP)
int net_cls_subsys_id = -1;
EXPORT_SYMBOL_GPL(net_cls_subsys_id);
--
1.7.3.4
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* [PATCH 08/14] netvm: Allow skb allocation to use PFMEMALLOC reserves
From: Mel Gorman @ 2011-06-09 8:02 UTC (permalink / raw)
To: Andrew Morton
Cc: Linux-MM, Linux-Netdev, LKML, David Miller, Neil Brown,
Peter Zijlstra, Mel Gorman
In-Reply-To: <1307606573-24704-1-git-send-email-mgorman@suse.de>
Change the skb allocation API to indicate RX usage and use this to fall back
to the PFMEMALLOC reserve when needed. SKBs allocated from the reserve are
tagged in skb->pfmemalloc. If an SKB is allocated from the reserve and
the socket is later found to be unrelated to page reclaim, the packet is
dropped so that the memory remains available for page reclaim. Network
protocols are expected to recover from this packet loss.
[a.p.zijlstra@chello.nl: Ideas taken from various patches]
Signed-off-by: Mel Gorman <mgorman@suse.de>
---
include/linux/gfp.h | 3 ++
include/linux/skbuff.h | 19 ++++++++--
include/net/sock.h | 6 +++
mm/internal.h | 3 --
net/core/filter.c | 8 ++++
net/core/skbuff.c | 95 ++++++++++++++++++++++++++++++++++++++++--------
net/core/sock.c | 4 ++
7 files changed, 116 insertions(+), 22 deletions(-)
diff --git a/include/linux/gfp.h b/include/linux/gfp.h
index faefba9..3cc24bb 100644
--- a/include/linux/gfp.h
+++ b/include/linux/gfp.h
@@ -375,6 +375,9 @@ void drain_local_pages(void *dummy);
extern gfp_t gfp_allowed_mask;
+/* Returns true if the gfp_mask allows use of ALLOC_NO_WATERMARK */
+bool gfp_pfmemalloc_allowed(gfp_t gfp_mask);
+
extern void pm_restrict_gfp_mask(void);
extern void pm_restore_gfp_mask(void);
diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h
index e8b78ce..ff8918f 100644
--- a/include/linux/skbuff.h
+++ b/include/linux/skbuff.h
@@ -396,6 +396,7 @@ struct sk_buff {
#ifdef CONFIG_IPV6_NDISC_NODETYPE
__u8 ndisc_nodetype:2;
#endif
+ __u8 pfmemalloc:1;
__u8 ooo_okay:1;
kmemcheck_bitfield_end(flags2);
@@ -434,6 +435,15 @@ struct sk_buff {
#include <asm/system.h>
+#define SKB_ALLOC_FCLONE 0x01
+#define SKB_ALLOC_RX 0x02
+
+/* Returns true if the skb was allocated from PFMEMALLOC reserves */
+static inline bool skb_pfmemalloc(struct sk_buff *skb)
+{
+ return unlikely(skb->pfmemalloc);
+}
+
/*
* skb might have a dst pointer attached, refcounted or not.
* _skb_refdst low order bit is set if refcount was _not_ taken
@@ -491,7 +501,7 @@ extern void kfree_skb(struct sk_buff *skb);
extern void consume_skb(struct sk_buff *skb);
extern void __kfree_skb(struct sk_buff *skb);
extern struct sk_buff *__alloc_skb(unsigned int size,
- gfp_t priority, int fclone, int node);
+ gfp_t priority, int flags, int node);
static inline struct sk_buff *alloc_skb(unsigned int size,
gfp_t priority)
{
@@ -501,7 +511,7 @@ static inline struct sk_buff *alloc_skb(unsigned int size,
static inline struct sk_buff *alloc_skb_fclone(unsigned int size,
gfp_t priority)
{
- return __alloc_skb(size, priority, 1, NUMA_NO_NODE);
+ return __alloc_skb(size, priority, SKB_ALLOC_FCLONE, NUMA_NO_NODE);
}
extern bool skb_recycle_check(struct sk_buff *skb, int skb_size);
@@ -1527,7 +1537,8 @@ static inline void __skb_queue_purge(struct sk_buff_head *list)
static inline struct sk_buff *__dev_alloc_skb(unsigned int length,
gfp_t gfp_mask)
{
- struct sk_buff *skb = alloc_skb(length + NET_SKB_PAD, gfp_mask);
+ struct sk_buff *skb = __alloc_skb(length + NET_SKB_PAD, gfp_mask,
+ SKB_ALLOC_RX, NUMA_NO_NODE);
if (likely(skb))
skb_reserve(skb, NET_SKB_PAD);
return skb;
@@ -1578,7 +1589,7 @@ static inline struct sk_buff *netdev_alloc_skb_ip_align(struct net_device *dev,
*/
static inline struct page *__netdev_alloc_page(struct net_device *dev, gfp_t gfp_mask)
{
- return alloc_pages_node(NUMA_NO_NODE, gfp_mask, 0);
+ return alloc_pages_node(NUMA_NO_NODE, gfp_mask | __GFP_MEMALLOC, 0);
}
/**
diff --git a/include/net/sock.h b/include/net/sock.h
index 046bc97..e3aaa88 100644
--- a/include/net/sock.h
+++ b/include/net/sock.h
@@ -586,6 +586,12 @@ static inline int sock_flag(struct sock *sk, enum sock_flags flag)
return test_bit(flag, &sk->sk_flags);
}
+extern atomic_t memalloc_socks;
+static inline int sk_memalloc_socks(void)
+{
+ return atomic_read(&memalloc_socks);
+}
+
static inline gfp_t sk_allocation(struct sock *sk, gfp_t gfp_mask)
{
return gfp_mask | (sk->sk_allocation & __GFP_MEMALLOC);
diff --git a/mm/internal.h b/mm/internal.h
index a520f3b..d071d380 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -193,9 +193,6 @@ static inline struct page *mem_map_next(struct page *iter,
#define __paginginit __init
#endif
-/* Returns true if the gfp_mask allows use of ALLOC_NO_WATERMARK */
-bool gfp_pfmemalloc_allowed(gfp_t gfp_mask);
-
/* Memory initialisation debug and verification */
enum mminit_level {
MMINIT_WARNING,
diff --git a/net/core/filter.c b/net/core/filter.c
index 36f975f..4ccf6f4 100644
--- a/net/core/filter.c
+++ b/net/core/filter.c
@@ -80,6 +80,14 @@ int sk_filter(struct sock *sk, struct sk_buff *skb)
int err;
struct sk_filter *filter;
+ /*
+ * If the skb was allocated from pfmemalloc reserves, only
+ * allow SOCK_MEMALLOC sockets to use it as this socket is
+ * helping free memory
+ */
+ if (skb_pfmemalloc(skb) && !sock_flag(sk, SOCK_MEMALLOC))
+ return -ENOMEM;
+
err = security_sock_rcv_skb(sk, skb);
if (err)
return err;
diff --git a/net/core/skbuff.c b/net/core/skbuff.c
index 46cbd28..d930b92 100644
--- a/net/core/skbuff.c
+++ b/net/core/skbuff.c
@@ -147,6 +147,43 @@ static void skb_under_panic(struct sk_buff *skb, int sz, void *here)
BUG();
}
+
+/*
+ * kmalloc_reserve is a wrapper around kmalloc_node_track_caller that tells
+ * the caller if emergency pfmemalloc reserves are being used. If it is and
+ * the socket is later found to be SOCK_MEMALLOC then PFMEMALLOC reserves
+ * may be used. Otherwise, the packet data may be discarded until enough
+ * memory is free
+ */
+#define kmalloc_reserve(size, gfp, node, pfmemalloc) \
+ __kmalloc_reserve(size, gfp, node, _RET_IP_, pfmemalloc)
+void *__kmalloc_reserve(size_t size, gfp_t flags, int node, unsigned long ip,
+ bool *pfmemalloc)
+{
+ void *obj;
+ bool ret_pfmemalloc = false;
+
+ /*
+ * Try a regular allocation, when that fails and we're not entitled
+ * to the reserves, fail.
+ */
+ obj = kmalloc_node_track_caller(size,
+ flags | __GFP_NOMEMALLOC | __GFP_NOWARN,
+ node);
+ if (obj || !(gfp_pfmemalloc_allowed(flags)))
+ goto out;
+
+ /* Try again but now we are using pfmemalloc reserves */
+ ret_pfmemalloc = true;
+ obj = kmalloc_node_track_caller(size, flags, node);
+
+out:
+ if (pfmemalloc)
+ *pfmemalloc = ret_pfmemalloc;
+
+ return obj;
+}
+
/* Allocate a new skbuff. We do this ourselves so we can fill in a few
* 'private' fields and also do memory statistics to find all the
* [BEEP] leaks.
@@ -157,8 +194,10 @@ static void skb_under_panic(struct sk_buff *skb, int sz, void *here)
* __alloc_skb - allocate a network buffer
* @size: size to allocate
* @gfp_mask: allocation mask
- * @fclone: allocate from fclone cache instead of head cache
- * and allocate a cloned (child) skb
+ * @flags: If SKB_ALLOC_FCLONE is set, allocate from fclone cache
+ * instead of head cache and allocate a cloned (child) skb.
+ * If SKB_ALLOC_RX is set, __GFP_MEMALLOC will be used for
+ * allocations in case the data is required for writeback
* @node: numa node to allocate memory on
*
* Allocate a new &sk_buff. The returned buffer has no headroom and a
@@ -169,14 +208,19 @@ static void skb_under_panic(struct sk_buff *skb, int sz, void *here)
* %GFP_ATOMIC.
*/
struct sk_buff *__alloc_skb(unsigned int size, gfp_t gfp_mask,
- int fclone, int node)
+ int flags, int node)
{
struct kmem_cache *cache;
struct skb_shared_info *shinfo;
struct sk_buff *skb;
u8 *data;
+ bool pfmemalloc;
+
+ cache = (flags & SKB_ALLOC_FCLONE)
+ ? skbuff_fclone_cache : skbuff_head_cache;
- cache = fclone ? skbuff_fclone_cache : skbuff_head_cache;
+ if (sk_memalloc_socks() && (flags & SKB_ALLOC_RX))
+ gfp_mask |= __GFP_MEMALLOC;
/* Get the HEAD */
skb = kmem_cache_alloc_node(cache, gfp_mask & ~__GFP_DMA, node);
@@ -185,8 +229,8 @@ struct sk_buff *__alloc_skb(unsigned int size, gfp_t gfp_mask,
prefetchw(skb);
size = SKB_DATA_ALIGN(size);
- data = kmalloc_node_track_caller(size + sizeof(struct skb_shared_info),
- gfp_mask, node);
+ data = kmalloc_reserve(size + sizeof(struct skb_shared_info),
+ gfp_mask, node, &pfmemalloc);
if (!data)
goto nodata;
prefetchw(data + size);
@@ -197,6 +241,7 @@ struct sk_buff *__alloc_skb(unsigned int size, gfp_t gfp_mask,
* the tail pointer in struct sk_buff!
*/
memset(skb, 0, offsetof(struct sk_buff, tail));
+ skb->pfmemalloc = pfmemalloc;
skb->truesize = size + sizeof(struct sk_buff);
atomic_set(&skb->users, 1);
skb->head = data;
@@ -213,7 +258,7 @@ struct sk_buff *__alloc_skb(unsigned int size, gfp_t gfp_mask,
atomic_set(&shinfo->dataref, 1);
kmemcheck_annotate_variable(shinfo->destructor_arg);
- if (fclone) {
+ if (flags & SKB_ALLOC_FCLONE) {
struct sk_buff *child = skb + 1;
atomic_t *fclone_ref = (atomic_t *) (child + 1);
@@ -223,6 +268,7 @@ struct sk_buff *__alloc_skb(unsigned int size, gfp_t gfp_mask,
atomic_set(fclone_ref, 1);
child->fclone = SKB_FCLONE_UNAVAILABLE;
+ child->pfmemalloc = pfmemalloc;
}
out:
return skb;
@@ -251,7 +297,8 @@ struct sk_buff *__netdev_alloc_skb(struct net_device *dev,
{
struct sk_buff *skb;
- skb = __alloc_skb(length + NET_SKB_PAD, gfp_mask, 0, NUMA_NO_NODE);
+ skb = __alloc_skb(length + NET_SKB_PAD, gfp_mask,
+ SKB_ALLOC_RX, NUMA_NO_NODE);
if (likely(skb)) {
skb_reserve(skb, NET_SKB_PAD);
skb->dev = dev;
@@ -527,6 +574,7 @@ static void __copy_skb_header(struct sk_buff *new, const struct sk_buff *old)
#if defined(CONFIG_IP_VS) || defined(CONFIG_IP_VS_MODULE)
new->ipvs_property = old->ipvs_property;
#endif
+ new->pfmemalloc = old->pfmemalloc;
new->protocol = old->protocol;
new->mark = old->mark;
new->skb_iif = old->skb_iif;
@@ -621,6 +669,9 @@ struct sk_buff *skb_clone(struct sk_buff *skb, gfp_t gfp_mask)
n->fclone = SKB_FCLONE_CLONE;
atomic_inc(fclone_ref);
} else {
+ if (skb_pfmemalloc(skb))
+ gfp_mask |= __GFP_MEMALLOC;
+
n = kmem_cache_alloc(skbuff_head_cache, gfp_mask);
if (!n)
return NULL;
@@ -657,6 +708,13 @@ static void copy_skb_header(struct sk_buff *new, const struct sk_buff *old)
skb_shinfo(new)->gso_type = skb_shinfo(old)->gso_type;
}
+static inline int skb_alloc_rx_flag(const struct sk_buff *skb)
+{
+ if (skb_pfmemalloc((struct sk_buff *)skb))
+ return SKB_ALLOC_RX;
+ return 0;
+}
+
/**
* skb_copy - create private copy of an sk_buff
* @skb: buffer to copy
@@ -678,7 +736,8 @@ struct sk_buff *skb_copy(const struct sk_buff *skb, gfp_t gfp_mask)
{
int headerlen = skb_headroom(skb);
unsigned int size = (skb_end_pointer(skb) - skb->head) + skb->data_len;
- struct sk_buff *n = alloc_skb(size, gfp_mask);
+ struct sk_buff *n = __alloc_skb(size, gfp_mask,
+ skb_alloc_rx_flag(skb), NUMA_NO_NODE);
if (!n)
return NULL;
@@ -712,7 +771,8 @@ EXPORT_SYMBOL(skb_copy);
struct sk_buff *pskb_copy(struct sk_buff *skb, gfp_t gfp_mask)
{
unsigned int size = skb_end_pointer(skb) - skb->head;
- struct sk_buff *n = alloc_skb(size, gfp_mask);
+ struct sk_buff *n = __alloc_skb(size, gfp_mask,
+ skb_alloc_rx_flag(skb), NUMA_NO_NODE);
if (!n)
goto out;
@@ -803,7 +863,10 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
goto adjust_others;
}
- data = kmalloc(size + sizeof(struct skb_shared_info), gfp_mask);
+ if (skb_pfmemalloc(skb))
+ gfp_mask |= __GFP_MEMALLOC;
+ data = kmalloc_reserve(size + sizeof(struct skb_shared_info), gfp_mask,
+ NUMA_NO_NODE, NULL);
if (!data)
goto nodata;
@@ -904,8 +967,9 @@ struct sk_buff *skb_copy_expand(const struct sk_buff *skb,
/*
* Allocate the copy buffer
*/
- struct sk_buff *n = alloc_skb(newheadroom + skb->len + newtailroom,
- gfp_mask);
+ struct sk_buff *n = __alloc_skb(newheadroom + skb->len + newtailroom,
+ gfp_mask, skb_alloc_rx_flag(skb),
+ NUMA_NO_NODE);
int oldheadroom = skb_headroom(skb);
int head_copy_len, head_copy_off;
int off;
@@ -2552,8 +2616,9 @@ struct sk_buff *skb_segment(struct sk_buff *skb, u32 features)
skb_release_head_state(nskb);
__skb_push(nskb, doffset);
} else {
- nskb = alloc_skb(hsize + doffset + headroom,
- GFP_ATOMIC);
+ nskb = __alloc_skb(hsize + doffset + headroom,
+ GFP_ATOMIC, skb_alloc_rx_flag(skb),
+ NUMA_NO_NODE);
if (unlikely(!nskb))
goto err;
diff --git a/net/core/sock.c b/net/core/sock.c
index c685eda..8308609 100644
--- a/net/core/sock.c
+++ b/net/core/sock.c
@@ -219,6 +219,8 @@ __u32 sysctl_rmem_default __read_mostly = SK_RMEM_MAX;
int sysctl_optmem_max __read_mostly = sizeof(unsigned long)*(2*UIO_MAXIOV+512);
EXPORT_SYMBOL(sysctl_optmem_max);
+atomic_t memalloc_socks __read_mostly;
+
/**
* sk_set_memalloc - sets %SOCK_MEMALLOC
* @sk: socket to set it on
@@ -231,6 +233,7 @@ void sk_set_memalloc(struct sock *sk)
{
sock_set_flag(sk, SOCK_MEMALLOC);
sk->sk_allocation |= __GFP_MEMALLOC;
+ atomic_inc(&memalloc_socks);
}
EXPORT_SYMBOL_GPL(sk_set_memalloc);
@@ -238,6 +241,7 @@ void sk_clear_memalloc(struct sock *sk)
{
sock_reset_flag(sk, SOCK_MEMALLOC);
sk->sk_allocation &= ~__GFP_MEMALLOC;
+ atomic_dec(&memalloc_socks);
}
EXPORT_SYMBOL_GPL(sk_clear_memalloc);
--
1.7.3.4
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* [PATCH 09/14] netvm: Propagate page->pfmemalloc to skb
From: Mel Gorman @ 2011-06-09 8:02 UTC (permalink / raw)
To: Andrew Morton
Cc: Linux-MM, Linux-Netdev, LKML, David Miller, Neil Brown,
Peter Zijlstra, Mel Gorman
In-Reply-To: <1307606573-24704-1-git-send-email-mgorman@suse.de>
The skb->pfmemalloc flag gets set to true iff during the slab
allocation of data in __alloc_skb that the the PFMEMALLOC reserves
were used. If the packet is fragmented, it is possible that pages
will be allocated from the PFMEMALLOC reserve without propagating
this information to the skb. This patch propagates page->pfmemalloc
from pages allocated for fragments to the skb.
Signed-off-by: Mel Gorman <mgorman@suse.de>
---
include/linux/skbuff.h | 2 ++
1 files changed, 2 insertions(+), 0 deletions(-)
diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h
index ff8918f..f924ca8 100644
--- a/include/linux/skbuff.h
+++ b/include/linux/skbuff.h
@@ -1124,6 +1124,8 @@ static inline void skb_fill_page_desc(struct sk_buff *skb, int i,
{
skb_frag_t *frag = &skb_shinfo(skb)->frags[i];
+ if (page->pfmemalloc)
+ skb->pfmemalloc = true;
frag->page = page;
frag->page_offset = off;
frag->size = size;
--
1.7.3.4
^ permalink raw reply related
* [PATCH 10/14] netvm: Set PF_MEMALLOC as appropriate during SKB processing
From: Mel Gorman @ 2011-06-09 8:02 UTC (permalink / raw)
To: Andrew Morton
Cc: Linux-MM, Linux-Netdev, LKML, David Miller, Neil Brown,
Peter Zijlstra, Mel Gorman
In-Reply-To: <1307606573-24704-1-git-send-email-mgorman@suse.de>
In order to make sure pfmemalloc packets receive all memory
needed to proceed, ensure processing of pfmemalloc SKBs happens
under PF_MEMALLOC. This is limited to a subset of protocols that
are expected to be used for writing to swap. Taps are not allowed to
use PF_MEMALLOC as these are expected to communicate with userspace
processes which could be paged out.
[a.p.zijlstra@chello.nl: Ideas taken from various patches]
[jslaby@suse.cz: Lock imbalance fix]
Signed-off-by: Mel Gorman <mgorman@suse.de>
---
include/net/sock.h | 5 +++++
net/core/dev.c | 48 ++++++++++++++++++++++++++++++++++++++++++++----
net/core/sock.c | 16 ++++++++++++++++
3 files changed, 65 insertions(+), 4 deletions(-)
diff --git a/include/net/sock.h b/include/net/sock.h
index e3aaa88..e928880 100644
--- a/include/net/sock.h
+++ b/include/net/sock.h
@@ -668,8 +668,13 @@ static inline __must_check int sk_add_backlog(struct sock *sk, struct sk_buff *s
return 0;
}
+extern int __sk_backlog_rcv(struct sock *sk, struct sk_buff *skb);
+
static inline int sk_backlog_rcv(struct sock *sk, struct sk_buff *skb)
{
+ if (skb_pfmemalloc(skb))
+ return __sk_backlog_rcv(sk, skb);
+
return sk->sk_backlog_rcv(sk, skb);
}
diff --git a/net/core/dev.c b/net/core/dev.c
index 9393078..c228286 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -3089,6 +3089,23 @@ void netdev_rx_handler_unregister(struct net_device *dev)
}
EXPORT_SYMBOL_GPL(netdev_rx_handler_unregister);
+/*
+ * Limit which protocols can use the PFMEMALLOC reserves to those that are
+ * expected to be used for communication with swap.
+ */
+static bool skb_pfmemalloc_protocol(struct sk_buff *skb)
+{
+ switch (skb->protocol) {
+ case __constant_htons(ETH_P_ARP):
+ case __constant_htons(ETH_P_IP):
+ case __constant_htons(ETH_P_IPV6):
+ case __constant_htons(ETH_P_8021Q):
+ return true;
+ default:
+ return false;
+ }
+}
+
static int __netif_receive_skb(struct sk_buff *skb)
{
struct packet_type *ptype, *pt_prev;
@@ -3098,15 +3115,28 @@ static int __netif_receive_skb(struct sk_buff *skb)
bool deliver_exact = false;
int ret = NET_RX_DROP;
__be16 type;
+ unsigned long pflags = current->flags;
if (!netdev_tstamp_prequeue)
net_timestamp_check(skb);
trace_netif_receive_skb(skb);
+ /*
+ * PFMEMALLOC skbs are special, they should
+ * - be delivered to SOCK_MEMALLOC sockets only
+ * - stay away from userspace
+ * - have bounded memory usage
+ *
+ * Use PF_MEMALLOC as this saves us from propagating the allocation
+ * context down to all allocation sites.
+ */
+ if (skb_pfmemalloc(skb))
+ current->flags |= PF_MEMALLOC;
+
/* if we've gotten here through NAPI, check netpoll */
if (netpoll_receive_skb(skb))
- return NET_RX_DROP;
+ goto out;
if (!skb->skb_iif)
skb->skb_iif = skb->dev->ifindex;
@@ -3137,6 +3167,9 @@ another_round:
}
#endif
+ if (skb_pfmemalloc(skb))
+ goto skip_taps;
+
list_for_each_entry_rcu(ptype, &ptype_all, list) {
if (!ptype->dev || ptype->dev == skb->dev) {
if (pt_prev)
@@ -3145,13 +3178,17 @@ another_round:
}
}
+skip_taps:
#ifdef CONFIG_NET_CLS_ACT
skb = handle_ing(skb, &pt_prev, &ret, orig_dev);
if (!skb)
- goto out;
+ goto unlock;
ncls:
#endif
+ if (skb_pfmemalloc(skb) && !skb_pfmemalloc_protocol(skb))
+ goto drop;
+
rx_handler = rcu_dereference(skb->dev->rx_handler);
if (rx_handler) {
if (pt_prev) {
@@ -3160,7 +3197,7 @@ ncls:
}
switch (rx_handler(&skb)) {
case RX_HANDLER_CONSUMED:
- goto out;
+ goto unlock;
case RX_HANDLER_ANOTHER:
goto another_round;
case RX_HANDLER_EXACT:
@@ -3202,6 +3239,7 @@ ncls:
if (pt_prev) {
ret = pt_prev->func(skb, skb->dev, pt_prev, orig_dev);
} else {
+drop:
atomic_long_inc(&skb->dev->rx_dropped);
kfree_skb(skb);
/* Jamal, now you will not able to escape explaining
@@ -3210,8 +3248,10 @@ ncls:
ret = NET_RX_DROP;
}
-out:
+unlock:
rcu_read_unlock();
+out:
+ tsk_restore_flags(current, pflags, PF_MEMALLOC);
return ret;
}
diff --git a/net/core/sock.c b/net/core/sock.c
index 8308609..ac36807 100644
--- a/net/core/sock.c
+++ b/net/core/sock.c
@@ -245,6 +245,22 @@ void sk_clear_memalloc(struct sock *sk)
}
EXPORT_SYMBOL_GPL(sk_clear_memalloc);
+int __sk_backlog_rcv(struct sock *sk, struct sk_buff *skb)
+{
+ int ret;
+ unsigned long pflags = current->flags;
+
+ /* these should have been dropped before queueing */
+ BUG_ON(!sock_flag(sk, SOCK_MEMALLOC));
+
+ current->flags |= PF_MEMALLOC;
+ ret = sk->sk_backlog_rcv(sk, skb);
+ tsk_restore_flags(current, pflags, PF_MEMALLOC);
+
+ return ret;
+}
+EXPORT_SYMBOL(__sk_backlog_rcv);
+
#if defined(CONFIG_CGROUPS) && !defined(CONFIG_NET_CLS_CGROUP)
int net_cls_subsys_id = -1;
EXPORT_SYMBOL_GPL(net_cls_subsys_id);
--
1.7.3.4
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* [PATCH 11/14] mm: Micro-optimise slab to avoid a function call
From: Mel Gorman @ 2011-06-09 8:02 UTC (permalink / raw)
To: Andrew Morton
Cc: Linux-MM, Linux-Netdev, LKML, David Miller, Neil Brown,
Peter Zijlstra, Mel Gorman
In-Reply-To: <1307606573-24704-1-git-send-email-mgorman@suse.de>
Getting and putting objects in SLAB currently requires a function call
but the bulk of the work is related to PFMEMALLOC reserves which are
only consumed when network-backed storage is critical. Use an inline
function to determine if the function call is required.
Signed-off-by: Mel Gorman <mgorman@suse.de>
---
mm/slab.c | 28 ++++++++++++++++++++++++++--
1 files changed, 26 insertions(+), 2 deletions(-)
diff --git a/mm/slab.c b/mm/slab.c
index 708da2f..3f49768 100644
--- a/mm/slab.c
+++ b/mm/slab.c
@@ -117,6 +117,8 @@
#include <linux/memory.h>
#include <linux/prefetch.h>
+#include <net/sock.h>
+
#include <asm/cacheflush.h>
#include <asm/tlbflush.h>
#include <asm/page.h>
@@ -945,7 +947,7 @@ static void check_ac_pfmemalloc(struct kmem_cache *cachep,
ac->pfmemalloc = false;
}
-static void *ac_get_obj(struct kmem_cache *cachep, struct array_cache *ac,
+static void *__ac_get_obj(struct kmem_cache *cachep, struct array_cache *ac,
gfp_t flags, bool force_refill)
{
int i;
@@ -992,7 +994,20 @@ static void *ac_get_obj(struct kmem_cache *cachep, struct array_cache *ac,
return objp;
}
-static void ac_put_obj(struct kmem_cache *cachep, struct array_cache *ac,
+static inline void *ac_get_obj(struct kmem_cache *cachep,
+ struct array_cache *ac, gfp_t flags, bool force_refill)
+{
+ void *objp;
+
+ if (unlikely(sk_memalloc_socks()))
+ objp = __ac_get_obj(cachep, ac, flags, force_refill);
+ else
+ objp = ac->entry[--ac->avail];
+
+ return objp;
+}
+
+static void *__ac_put_obj(struct kmem_cache *cachep, struct array_cache *ac,
void *objp)
{
struct slab *slabp;
@@ -1005,6 +1020,15 @@ static void ac_put_obj(struct kmem_cache *cachep, struct array_cache *ac,
set_obj_pfmemalloc(&objp);
}
+ return objp;
+}
+
+static inline void ac_put_obj(struct kmem_cache *cachep, struct array_cache *ac,
+ void *objp)
+{
+ if (unlikely(sk_memalloc_socks()))
+ objp = __ac_put_obj(cachep, ac, objp);
+
ac->entry[ac->avail++] = objp;
}
--
1.7.3.4
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox