* Re: [patch 4/4] PPP synchronous tty: convert dead_sem to completion
From: David Miller @ 2007-12-14 19:33 UTC (permalink / raw)
To: akpm; +Cc: netdev, matthias.kaehlcke, paulus
In-Reply-To: <200712140002.lBE02cF7025497@imap1.linux-foundation.org>
From: akpm@linux-foundation.org
Date: Thu, 13 Dec 2007 16:02:37 -0800
> From: Matthias Kaehlcke <matthias.kaehlcke@gmail.com>
>
> PPP synchronous tty channel driver: convert the semaphore dead_sem to a
> completion
>
> Signed-off-by: Matthias Kaehlcke <matthias.kaehlcke@gmail.com>
> Cc: Paul Mackerras <paulus@samba.org>
> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Applied to net-2.6.25, thanks.
^ permalink raw reply
* Re: [patch 3/4] tipc: fix semaphore handling
From: David Miller @ 2007-12-14 19:31 UTC (permalink / raw)
To: akpm; +Cc: netdev, allan.stephens, jon.maloy, kjwinchester, per.liden,
stable
In-Reply-To: <200712140002.lBE02b3G025494@imap1.linux-foundation.org>
From: akpm@linux-foundation.org
Date: Thu, 13 Dec 2007 16:02:36 -0800
> From: Andrew Morton <akpm@linux-foundation.org>
>
> As noted by Kevin, tipc's release() does down_interruptible() and ignores the
> return value. So if signal_pending() we'll end up doing up() on a non-downed
> semaphore. Fix.
>
> Cc: Kevin Winchester <kjwinchester@gmail.com>
> Cc: Per Liden <per.liden@ericsson.com>
> Cc: Jon Maloy <jon.maloy@ericsson.com>
> Cc: Allan Stephens <allan.stephens@windriver.com>
> Cc: "David S. Miller" <davem@davemloft.net>
> Cc: <stable@kernel.org>
> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
This is already in my net-2.6 tree, but thanks for resubmitting
anyways :)
^ permalink raw reply
* Re: [patch 2/4] net: use mutex_is_locked() for ASSERT_RTNL()
From: David Miller @ 2007-12-14 19:30 UTC (permalink / raw)
To: akpm; +Cc: netdev
In-Reply-To: <200712140002.lBE02aGO025491@imap1.linux-foundation.org>
From: akpm@linux-foundation.org
Date: Thu, 13 Dec 2007 16:02:36 -0800
> From: Andrew Morton <akpm@linux-foundation.org>
>
> ASSERT_RTNL() uses mutex_trylock(), but it's better to use mutex_is_locked().
>
> Make that change, and remove rtnl_trylock() altogether.
>
> (not tested yet!)
>
> Cc: "David S. Miller" <davem@davemloft.net>
> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
NACK, as explained please remove this until the replacement
doesn't remove valid checks which are done currently.
^ permalink raw reply
* Re: [patch 1/4] Updates to nfsroot documentation
From: David Miller @ 2007-12-14 19:30 UTC (permalink / raw)
To: akpm; +Cc: netdev, apw, ak, horms
In-Reply-To: <200712140002.lBE02XWr025488@imap1.linux-foundation.org>
From: akpm@linux-foundation.org
Date: Thu, 13 Dec 2007 16:02:33 -0800
> From: Amos Waterland <apw@us.ibm.com>
>
> The difference between ip=off and ip=::::::off has been a cause of much
> confusion. Document how each behaves, and do not contradict ourselves by
> saying that "off" is the default when in fact "any" is the default and is
> descibed as being so lower in the file.
>
> Signed-off-by: Amos Waterland <apw@us.ibm.com>
> Cc: Simon Horman <horms@verge.net.au>
> Cc: Andi Kleen <ak@suse.de>
> Cc: "David S. Miller" <davem@davemloft.net>
> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Applied.
^ permalink raw reply
* Re: [PATCHES 0/3]: DCCP patches for 2.6.25
From: David Miller @ 2007-12-14 19:29 UTC (permalink / raw)
To: acme; +Cc: netdev, dccp
In-Reply-To: <1197596522-14600-1-git-send-email-acme@redhat.com>
From: Arnaldo Carvalho de Melo <acme@redhat.com>
Date: Thu, 13 Dec 2007 23:41:59 -0200
> Please consider pulling from:
>
> master.kernel.org:/pub/scm/linux/kernel/git/acme/net-2.6.25
Pulled, but could you please reformat Gerrit's changelog entries in
the future? They have these 80+ long lines which are painful to read
in ascii email clients and in terminal output.
I'll do this by hand during my next rebase for this case, but I will
push back when I see it again in future pull requests.
^ permalink raw reply
* Re: 2.6.24-rc5-mm1
From: David Miller @ 2007-12-14 19:26 UTC (permalink / raw)
To: herbert; +Cc: benjamin.thery, pierre.peiffer, akpm, linux-kernel, netdev
In-Reply-To: <20071214020807.GA12426@gondor.apana.org.au>
From: Herbert Xu <herbert@gondor.apana.org.au>
Date: Fri, 14 Dec 2007 10:08:07 +0800
> [UDP]: Move udp_stats_in6 into net/ipv4/udp.c
>
> Now that external users may increment the counters directly, we need to
> ensure that udp_stats_in6 is always available. Otherwise we'd either
> have to requrie the external users to be built as modules or ipv6 to be
> built-in.
>
> This isn't too bad because udp_stats_in6 is just a pair of pointers plus
> an EXPORT, e.g., just 40 (16 + 24) bytes on x86-64.
>
> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Applied.
^ permalink raw reply
* [PATCH 28/29] nfs: enable swap on NFS
From: Peter Zijlstra @ 2007-12-14 15:39 UTC (permalink / raw)
To: Linus Torvalds, Andrew Morton, linux-kernel, linux-mm, netdev,
trond.myklebust
Cc: Peter Zijlstra
In-Reply-To: <20071214153907.770251000@chello.nl>
[-- Attachment #1: nfs-swap_ops.patch --]
[-- Type: text/plain, Size: 10831 bytes --]
Implement all the new swapfile a_ops for NFS. This will set the NFS socket to
SOCK_MEMALLOC and run socket reconnect under PF_MEMALLOC as well as reset
SOCK_MEMALLOC before engaging the protocol ->connect() method.
PF_MEMALLOC should allow the allocation of struct socket and related objects
and the early (re)setting of SOCK_MEMALLOC should allow us to receive the
packets required for the TCP connection buildup.
(swapping continues over a server reset during heavy network traffic)
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
---
fs/Kconfig | 18 ++++++++++++
fs/nfs/file.c | 12 ++++++++
fs/nfs/write.c | 19 +++++++++++++
include/linux/nfs_fs.h | 2 +
include/linux/sunrpc/xprt.h | 5 ++-
net/sunrpc/sched.c | 9 ++++--
net/sunrpc/xprtsock.c | 63 ++++++++++++++++++++++++++++++++++++++++++++
7 files changed, 125 insertions(+), 3 deletions(-)
Index: linux-2.6/fs/nfs/file.c
===================================================================
--- linux-2.6.orig/fs/nfs/file.c
+++ linux-2.6/fs/nfs/file.c
@@ -371,6 +371,13 @@ static int nfs_launder_page(struct page
return nfs_wb_page(page_file_mapping(page)->host, page);
}
+#ifdef CONFIG_NFS_SWAP
+static int nfs_swapfile(struct address_space *mapping, int enable)
+{
+ return xs_swapper(NFS_CLIENT(mapping->host)->cl_xprt, enable);
+}
+#endif
+
const struct address_space_operations nfs_file_aops = {
.readpage = nfs_readpage,
.readpages = nfs_readpages,
@@ -385,6 +392,11 @@ const struct address_space_operations nf
.direct_IO = nfs_direct_IO,
#endif
.launder_page = nfs_launder_page,
+#ifdef CONFIG_NFS_SWAP
+ .swapfile = nfs_swapfile,
+ .swap_out = nfs_swap_out,
+ .swap_in = nfs_readpage,
+#endif
};
static int nfs_vm_page_mkwrite(struct vm_area_struct *vma, struct page *page)
Index: linux-2.6/fs/nfs/write.c
===================================================================
--- linux-2.6.orig/fs/nfs/write.c
+++ linux-2.6/fs/nfs/write.c
@@ -365,6 +365,25 @@ int nfs_writepage(struct page *page, str
return ret;
}
+int nfs_swap_out(struct file *file, struct page *page,
+ struct writeback_control *wbc)
+{
+ struct nfs_open_context *ctx = nfs_file_open_context(file);
+ int status;
+
+ status = nfs_writepage_setup(ctx, page, 0, nfs_page_length(page));
+ if (status < 0) {
+ nfs_set_pageerror(page);
+ goto out;
+ }
+
+ status = nfs_writepage_locked(page, wbc);
+
+out:
+ unlock_page(page);
+ return status;
+}
+
static int nfs_writepages_callback(struct page *page, struct writeback_control *wbc, void *data)
{
int ret;
Index: linux-2.6/include/linux/nfs_fs.h
===================================================================
--- linux-2.6.orig/include/linux/nfs_fs.h
+++ linux-2.6/include/linux/nfs_fs.h
@@ -413,6 +413,8 @@ extern int nfs_flush_incompatible(struc
extern int nfs_updatepage(struct file *, struct page *, unsigned int, unsigned int);
extern int nfs_writeback_done(struct rpc_task *, struct nfs_write_data *);
extern void nfs_writedata_release(void *);
+extern int nfs_swap_out(struct file *file, struct page *page,
+ struct writeback_control *wbc);
/*
* Try to write back everything synchronously (but check the
Index: linux-2.6/fs/Kconfig
===================================================================
--- linux-2.6.orig/fs/Kconfig
+++ linux-2.6/fs/Kconfig
@@ -1692,6 +1692,18 @@ config NFS_DIRECTIO
causes open() to return EINVAL if a file residing in NFS is
opened with the O_DIRECT flag.
+config NFS_SWAP
+ bool "Provide swap over NFS support"
+ default n
+ depends on NFS_FS
+ select SUNRPC_SWAP
+ help
+ This option enables swapon to work on files located on NFS mounts.
+
+ For more details, see Documentation/vm_deadlock.txt
+
+ If unsure, say N.
+
config NFSD
tristate "NFS server support"
depends on INET
@@ -1835,6 +1847,12 @@ config SUNRPC_BIND34
If unsure, say N to get traditional behavior (version 2 rpcbind
requests only).
+config SUNRPC_SWAP
+ def_bool n
+ depends on SUNRPC
+ select NETVM
+ select SWAP_FILE
+
config RPCSEC_GSS_KRB5
tristate "Secure RPC: Kerberos V mechanism (EXPERIMENTAL)"
depends on SUNRPC && EXPERIMENTAL
Index: linux-2.6/include/linux/sunrpc/xprt.h
===================================================================
--- linux-2.6.orig/include/linux/sunrpc/xprt.h
+++ linux-2.6/include/linux/sunrpc/xprt.h
@@ -143,7 +143,9 @@ struct rpc_xprt {
unsigned int max_reqs; /* total slots */
unsigned long state; /* transport state */
unsigned char shutdown : 1, /* being shut down */
- resvport : 1; /* use a reserved port */
+ resvport : 1, /* use a reserved port */
+ swapper : 1; /* we're swapping over this
+ transport */
unsigned int bind_index; /* bind function index */
/*
@@ -246,6 +248,7 @@ struct rpc_rqst * xprt_lookup_rqst(struc
void xprt_complete_rqst(struct rpc_task *task, int copied);
void xprt_release_rqst_cong(struct rpc_task *task);
void xprt_disconnect(struct rpc_xprt *xprt);
+int xs_swapper(struct rpc_xprt *xprt, int enable);
/*
* Reserved bit positions in xprt->state
Index: linux-2.6/net/sunrpc/sched.c
===================================================================
--- linux-2.6.orig/net/sunrpc/sched.c
+++ linux-2.6/net/sunrpc/sched.c
@@ -761,7 +761,10 @@ struct rpc_buffer {
void *rpc_malloc(struct rpc_task *task, size_t size)
{
struct rpc_buffer *buf;
- gfp_t gfp = RPC_IS_SWAPPER(task) ? GFP_ATOMIC : GFP_NOWAIT;
+ gfp_t gfp = GFP_NOWAIT;
+
+ if (RPC_IS_SWAPPER(task))
+ gfp |= __GFP_MEMALLOC;
size += sizeof(struct rpc_buffer);
if (size <= RPC_BUFFER_MAXSIZE)
@@ -816,6 +819,8 @@ void rpc_init_task(struct rpc_task *task
atomic_set(&task->tk_count, 1);
task->tk_client = clnt;
task->tk_flags = flags;
+ if (clnt->cl_xprt->swapper)
+ task->tk_flags |= RPC_TASK_SWAPPER;
task->tk_ops = tk_ops;
if (tk_ops->rpc_call_prepare != NULL)
task->tk_action = rpc_prepare_task;
@@ -852,7 +857,7 @@ void rpc_init_task(struct rpc_task *task
static struct rpc_task *
rpc_alloc_task(void)
{
- return (struct rpc_task *)mempool_alloc(rpc_task_mempool, GFP_NOFS);
+ return (struct rpc_task *)mempool_alloc(rpc_task_mempool, GFP_NOIO);
}
static void rpc_free_task(struct rcu_head *rcu)
Index: linux-2.6/net/sunrpc/xprtsock.c
===================================================================
--- linux-2.6.orig/net/sunrpc/xprtsock.c
+++ linux-2.6/net/sunrpc/xprtsock.c
@@ -1397,6 +1397,9 @@ static void xs_udp_finish_connecting(str
transport->sock = sock;
transport->inet = sk;
+ if (xprt->swapper)
+ sk_set_memalloc(sk);
+
write_unlock_bh(&sk->sk_callback_lock);
}
xs_udp_do_set_buffer_size(xprt);
@@ -1414,11 +1417,15 @@ static void xs_udp_connect_worker4(struc
container_of(work, struct sock_xprt, connect_worker.work);
struct rpc_xprt *xprt = &transport->xprt;
struct socket *sock = transport->sock;
+ unsigned long pflags = current->flags;
int err, status = -EIO;
if (xprt->shutdown || !xprt_bound(xprt))
goto out;
+ if (xprt->swapper)
+ current->flags |= PF_MEMALLOC;
+
/* Start by resetting any existing state */
xs_close(xprt);
@@ -1441,6 +1448,7 @@ static void xs_udp_connect_worker4(struc
out:
xprt_wake_pending_tasks(xprt, status);
xprt_clear_connecting(xprt);
+ tsk_restore_flags(current, pflags, PF_MEMALLOC);
}
/**
@@ -1455,11 +1463,15 @@ static void xs_udp_connect_worker6(struc
container_of(work, struct sock_xprt, connect_worker.work);
struct rpc_xprt *xprt = &transport->xprt;
struct socket *sock = transport->sock;
+ unsigned long pflags = current->flags;
int err, status = -EIO;
if (xprt->shutdown || !xprt_bound(xprt))
goto out;
+ if (xprt->swapper)
+ current->flags |= PF_MEMALLOC;
+
/* Start by resetting any existing state */
xs_close(xprt);
@@ -1482,6 +1494,7 @@ static void xs_udp_connect_worker6(struc
out:
xprt_wake_pending_tasks(xprt, status);
xprt_clear_connecting(xprt);
+ tsk_restore_flags(current, pflags, PF_MEMALLOC);
}
/*
@@ -1541,6 +1554,9 @@ static int xs_tcp_finish_connecting(stru
write_unlock_bh(&sk->sk_callback_lock);
}
+ if (xprt->swapper)
+ sk_set_memalloc(transport->inet);
+
/* Tell the socket layer to start connecting... */
xprt->stat.connect_count++;
xprt->stat.connect_start = jiffies;
@@ -1559,11 +1575,15 @@ static void xs_tcp_connect_worker4(struc
container_of(work, struct sock_xprt, connect_worker.work);
struct rpc_xprt *xprt = &transport->xprt;
struct socket *sock = transport->sock;
+ unsigned long pflags = current->flags;
int err, status = -EIO;
if (xprt->shutdown || !xprt_bound(xprt))
goto out;
+ if (xprt->swapper)
+ current->flags |= PF_MEMALLOC;
+
if (!sock) {
/* start from scratch */
if ((err = sock_create_kern(PF_INET, SOCK_STREAM, IPPROTO_TCP, &sock)) < 0) {
@@ -1606,6 +1626,7 @@ out:
xprt_wake_pending_tasks(xprt, status);
out_clear:
xprt_clear_connecting(xprt);
+ tsk_restore_flags(current, pflags, PF_MEMALLOC);
}
/**
@@ -1620,11 +1641,15 @@ static void xs_tcp_connect_worker6(struc
container_of(work, struct sock_xprt, connect_worker.work);
struct rpc_xprt *xprt = &transport->xprt;
struct socket *sock = transport->sock;
+ unsigned long pflags = current->flags;
int err, status = -EIO;
if (xprt->shutdown || !xprt_bound(xprt))
goto out;
+ if (xprt->swapper)
+ current->flags |= PF_MEMALLOC;
+
if (!sock) {
/* start from scratch */
if ((err = sock_create_kern(PF_INET6, SOCK_STREAM, IPPROTO_TCP, &sock)) < 0) {
@@ -1666,6 +1691,7 @@ out:
xprt_wake_pending_tasks(xprt, status);
out_clear:
xprt_clear_connecting(xprt);
+ tsk_restore_flags(current, pflags, PF_MEMALLOC);
}
/**
@@ -1985,6 +2011,43 @@ int init_socket_xprt(void)
return 0;
}
+#ifdef CONFIG_SUNRPC_SWAP
+#define RPC_BUF_RESERVE_PAGES \
+ kestimate_single(sizeof(struct rpc_rqst), GFP_KERNEL, RPC_MAX_SLOT_TABLE)
+#define RPC_RESERVE_PAGES (RPC_BUF_RESERVE_PAGES + TX_RESERVE_PAGES)
+
+/**
+ * xs_swapper - Tag this transport as being used for swap.
+ * @xprt: transport to tag
+ * @enable: enable/disable
+ *
+ */
+int xs_swapper(struct rpc_xprt *xprt, int enable)
+{
+ struct sock_xprt *transport = container_of(xprt, struct sock_xprt, xprt);
+ int err = 0;
+
+ if (enable) {
+ /*
+ * keep one extra sock reference so the reserve won't dip
+ * when the socket gets reconnected.
+ */
+ err = sk_adjust_memalloc(1, RPC_RESERVE_PAGES);
+ if (!err) {
+ sk_set_memalloc(transport->inet);
+ xprt->swapper = 1;
+ }
+ } else if (xprt->swapper) {
+ xprt->swapper = 0;
+ sk_clear_memalloc(transport->inet);
+ sk_adjust_memalloc(-1, -RPC_RESERVE_PAGES);
+ }
+
+ return err;
+}
+EXPORT_SYMBOL_GPL(xs_swapper);
+#endif
+
/**
* cleanup_socket_xprt - remove xprtsock's sysctls, unregister
*
--
^ permalink raw reply
* [PATCH 25/29] nfs: remove mempools
From: Peter Zijlstra @ 2007-12-14 15:39 UTC (permalink / raw)
To: Linus Torvalds, Andrew Morton, linux-kernel, linux-mm, netdev,
trond.myklebust
Cc: Peter Zijlstra
In-Reply-To: <20071214153907.770251000@chello.nl>
[-- Attachment #1: nfs-no-mempool.patch --]
[-- Type: text/plain, Size: 4919 bytes --]
With the introduction of the shared dirty page accounting in .19, NFS should
not be able to surpise the VM with all dirty pages. Thus it should always be
able to free some memory. Hence no more need for mempools.
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
---
fs/nfs/read.c | 15 +++------------
fs/nfs/write.c | 27 +++++----------------------
2 files changed, 8 insertions(+), 34 deletions(-)
Index: linux-2.6/fs/nfs/read.c
===================================================================
--- linux-2.6.orig/fs/nfs/read.c
+++ linux-2.6/fs/nfs/read.c
@@ -33,13 +33,10 @@ static const struct rpc_call_ops nfs_rea
static const struct rpc_call_ops nfs_read_full_ops;
static struct kmem_cache *nfs_rdata_cachep;
-static mempool_t *nfs_rdata_mempool;
-
-#define MIN_POOL_READ (32)
struct nfs_read_data *nfs_readdata_alloc(unsigned int pagecount)
{
- struct nfs_read_data *p = mempool_alloc(nfs_rdata_mempool, GFP_NOFS);
+ struct nfs_read_data *p = kmem_cache_alloc(nfs_rdata_cachep, GFP_NOFS);
if (p) {
memset(p, 0, sizeof(*p));
@@ -50,7 +47,7 @@ struct nfs_read_data *nfs_readdata_alloc
else {
p->pagevec = kcalloc(pagecount, sizeof(struct page *), GFP_NOFS);
if (!p->pagevec) {
- mempool_free(p, nfs_rdata_mempool);
+ kmem_cache_free(nfs_rdata_cachep, p);
p = NULL;
}
}
@@ -63,7 +60,7 @@ static void nfs_readdata_rcu_free(struct
struct nfs_read_data *p = container_of(head, struct nfs_read_data, task.u.tk_rcu);
if (p && (p->pagevec != &p->page_array[0]))
kfree(p->pagevec);
- mempool_free(p, nfs_rdata_mempool);
+ kmem_cache_free(nfs_rdata_cachep, p);
}
static void nfs_readdata_free(struct nfs_read_data *rdata)
@@ -597,16 +594,10 @@ int __init nfs_init_readpagecache(void)
if (nfs_rdata_cachep == NULL)
return -ENOMEM;
- nfs_rdata_mempool = mempool_create_slab_pool(MIN_POOL_READ,
- nfs_rdata_cachep);
- if (nfs_rdata_mempool == NULL)
- return -ENOMEM;
-
return 0;
}
void nfs_destroy_readpagecache(void)
{
- mempool_destroy(nfs_rdata_mempool);
kmem_cache_destroy(nfs_rdata_cachep);
}
Index: linux-2.6/fs/nfs/write.c
===================================================================
--- linux-2.6.orig/fs/nfs/write.c
+++ linux-2.6/fs/nfs/write.c
@@ -28,9 +28,6 @@
#define NFSDBG_FACILITY NFSDBG_PAGECACHE
-#define MIN_POOL_WRITE (32)
-#define MIN_POOL_COMMIT (4)
-
/*
* Local function declarations
*/
@@ -44,12 +41,10 @@ static const struct rpc_call_ops nfs_wri
static const struct rpc_call_ops nfs_commit_ops;
static struct kmem_cache *nfs_wdata_cachep;
-static mempool_t *nfs_wdata_mempool;
-static mempool_t *nfs_commit_mempool;
struct nfs_write_data *nfs_commit_alloc(void)
{
- struct nfs_write_data *p = mempool_alloc(nfs_commit_mempool, GFP_NOFS);
+ struct nfs_write_data *p = kmem_cache_alloc(nfs_wdata_cachep, GFP_NOFS);
if (p) {
memset(p, 0, sizeof(*p));
@@ -63,7 +58,7 @@ static void nfs_commit_rcu_free(struct r
struct nfs_write_data *p = container_of(head, struct nfs_write_data, task.u.tk_rcu);
if (p && (p->pagevec != &p->page_array[0]))
kfree(p->pagevec);
- mempool_free(p, nfs_commit_mempool);
+ kmem_cache_free(nfs_wdata_cachep, p);
}
void nfs_commit_free(struct nfs_write_data *wdata)
@@ -73,7 +68,7 @@ void nfs_commit_free(struct nfs_write_da
struct nfs_write_data *nfs_writedata_alloc(unsigned int pagecount)
{
- struct nfs_write_data *p = mempool_alloc(nfs_wdata_mempool, GFP_NOFS);
+ struct nfs_write_data *p = kmem_cache_alloc(nfs_wdata_cachep, GFP_NOFS);
if (p) {
memset(p, 0, sizeof(*p));
@@ -84,7 +79,7 @@ struct nfs_write_data *nfs_writedata_all
else {
p->pagevec = kcalloc(pagecount, sizeof(struct page *), GFP_NOFS);
if (!p->pagevec) {
- mempool_free(p, nfs_wdata_mempool);
+ kmem_cache_free(nfs_wdata_cachep, p);
p = NULL;
}
}
@@ -97,7 +92,7 @@ static void nfs_writedata_rcu_free(struc
struct nfs_write_data *p = container_of(head, struct nfs_write_data, task.u.tk_rcu);
if (p && (p->pagevec != &p->page_array[0]))
kfree(p->pagevec);
- mempool_free(p, nfs_wdata_mempool);
+ kmem_cache_free(nfs_wdata_cachep, p);
}
static void nfs_writedata_free(struct nfs_write_data *wdata)
@@ -1474,16 +1469,6 @@ int __init nfs_init_writepagecache(void)
if (nfs_wdata_cachep == NULL)
return -ENOMEM;
- nfs_wdata_mempool = mempool_create_slab_pool(MIN_POOL_WRITE,
- nfs_wdata_cachep);
- if (nfs_wdata_mempool == NULL)
- return -ENOMEM;
-
- nfs_commit_mempool = mempool_create_slab_pool(MIN_POOL_COMMIT,
- nfs_wdata_cachep);
- if (nfs_commit_mempool == NULL)
- return -ENOMEM;
-
/*
* NFS congestion size, scale with available memory.
*
@@ -1509,8 +1494,6 @@ int __init nfs_init_writepagecache(void)
void nfs_destroy_writepagecache(void)
{
- mempool_destroy(nfs_commit_mempool);
- mempool_destroy(nfs_wdata_mempool);
kmem_cache_destroy(nfs_wdata_cachep);
}
--
^ permalink raw reply
* [PATCH 22/29] mm: prepare swap entry methods for use in page methods
From: Peter Zijlstra @ 2007-12-14 15:39 UTC (permalink / raw)
To: Linus Torvalds, Andrew Morton, linux-kernel, linux-mm, netdev,
trond.myklebust
Cc: Peter Zijlstra
In-Reply-To: <20071214153907.770251000@chello.nl>
[-- Attachment #1: mm-swap_entry_methods.patch --]
[-- Type: text/plain, Size: 5160 bytes --]
Move around the swap entry methods in preparation for use from
page methods.
Also provide a function to obtain the swap_info_struct backing
a swap cache page.
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
---
include/linux/mm.h | 8 +++++++
include/linux/swap.h | 49 ++++++++++++++++++++++++++++++++++++++++++++++++
include/linux/swapops.h | 44 -------------------------------------------
mm/swapfile.c | 1
4 files changed, 58 insertions(+), 44 deletions(-)
Index: linux-2.6/include/linux/mm.h
===================================================================
--- linux-2.6.orig/include/linux/mm.h
+++ linux-2.6/include/linux/mm.h
@@ -13,6 +13,7 @@
#include <linux/debug_locks.h>
#include <linux/mm_types.h>
#include <linux/security.h>
+#include <linux/swap.h>
struct mempolicy;
struct anon_vma;
@@ -600,6 +601,13 @@ static inline struct address_space *page
return mapping;
}
+static inline struct swap_info_struct *page_swap_info(struct page *page)
+{
+ swp_entry_t swap = { .val = page_private(page) };
+ BUG_ON(!PageSwapCache(page));
+ return get_swap_info_struct(swp_type(swap));
+}
+
static inline int PageAnon(struct page *page)
{
return ((unsigned long)page->mapping & PAGE_MAPPING_ANON) != 0;
Index: linux-2.6/include/linux/swap.h
===================================================================
--- linux-2.6.orig/include/linux/swap.h
+++ linux-2.6/include/linux/swap.h
@@ -80,6 +80,50 @@ typedef struct {
} swp_entry_t;
/*
+ * swapcache pages are stored in the swapper_space radix tree. We want to
+ * get good packing density in that tree, so the index should be dense in
+ * the low-order bits.
+ *
+ * We arrange the `type' and `offset' fields so that `type' is at the five
+ * high-order bits of the swp_entry_t and `offset' is right-aligned in the
+ * remaining bits.
+ *
+ * swp_entry_t's are *never* stored anywhere in their arch-dependent format.
+ */
+#define SWP_TYPE_SHIFT(e) (sizeof(e.val) * 8 - MAX_SWAPFILES_SHIFT)
+#define SWP_OFFSET_MASK(e) ((1UL << SWP_TYPE_SHIFT(e)) - 1)
+
+/*
+ * Store a type+offset into a swp_entry_t in an arch-independent format
+ */
+static inline swp_entry_t swp_entry(unsigned long type, pgoff_t offset)
+{
+ swp_entry_t ret;
+
+ ret.val = (type << SWP_TYPE_SHIFT(ret)) |
+ (offset & SWP_OFFSET_MASK(ret));
+ return ret;
+}
+
+/*
+ * Extract the `type' field from a swp_entry_t. The swp_entry_t is in
+ * arch-independent format
+ */
+static inline unsigned swp_type(swp_entry_t entry)
+{
+ return (entry.val >> SWP_TYPE_SHIFT(entry));
+}
+
+/*
+ * Extract the `offset' field from a swp_entry_t. The swp_entry_t is in
+ * arch-independent format
+ */
+static inline pgoff_t swp_offset(swp_entry_t entry)
+{
+ return entry.val & SWP_OFFSET_MASK(entry);
+}
+
+/*
* current->reclaim_state points to one of these when a task is running
* memory reclaim
*/
@@ -321,6 +365,11 @@ static inline struct page *lookup_swap_c
return NULL;
}
+static inline struct swap_info_struct *get_swap_info_struct(unsigned type)
+{
+ return NULL;
+}
+
#define can_share_swap_page(p) (page_mapcount(p) == 1)
static inline int move_to_swap_cache(struct page *page, swp_entry_t entry)
Index: linux-2.6/include/linux/swapops.h
===================================================================
--- linux-2.6.orig/include/linux/swapops.h
+++ linux-2.6/include/linux/swapops.h
@@ -1,47 +1,3 @@
-/*
- * swapcache pages are stored in the swapper_space radix tree. We want to
- * get good packing density in that tree, so the index should be dense in
- * the low-order bits.
- *
- * We arrange the `type' and `offset' fields so that `type' is at the five
- * high-order bits of the swp_entry_t and `offset' is right-aligned in the
- * remaining bits.
- *
- * swp_entry_t's are *never* stored anywhere in their arch-dependent format.
- */
-#define SWP_TYPE_SHIFT(e) (sizeof(e.val) * 8 - MAX_SWAPFILES_SHIFT)
-#define SWP_OFFSET_MASK(e) ((1UL << SWP_TYPE_SHIFT(e)) - 1)
-
-/*
- * Store a type+offset into a swp_entry_t in an arch-independent format
- */
-static inline swp_entry_t swp_entry(unsigned long type, pgoff_t offset)
-{
- swp_entry_t ret;
-
- ret.val = (type << SWP_TYPE_SHIFT(ret)) |
- (offset & SWP_OFFSET_MASK(ret));
- return ret;
-}
-
-/*
- * Extract the `type' field from a swp_entry_t. The swp_entry_t is in
- * arch-independent format
- */
-static inline unsigned swp_type(swp_entry_t entry)
-{
- return (entry.val >> SWP_TYPE_SHIFT(entry));
-}
-
-/*
- * Extract the `offset' field from a swp_entry_t. The swp_entry_t is in
- * arch-independent format
- */
-static inline pgoff_t swp_offset(swp_entry_t entry)
-{
- return entry.val & SWP_OFFSET_MASK(entry);
-}
-
/* check whether a pte points to a swap entry */
static inline int is_swap_pte(pte_t pte)
{
Index: linux-2.6/mm/swapfile.c
===================================================================
--- linux-2.6.orig/mm/swapfile.c
+++ linux-2.6/mm/swapfile.c
@@ -1795,6 +1795,7 @@ get_swap_info_struct(unsigned type)
{
return &swap_info[type];
}
+EXPORT_SYMBOL_GPL(get_swap_info_struct);
/*
* swap_lock prevents swap_map being freed. Don't grab an extra
--
^ permalink raw reply
* [PATCH 10/29] mm: memory reserve management
From: Peter Zijlstra @ 2007-12-14 15:39 UTC (permalink / raw)
To: Linus Torvalds, Andrew Morton, linux-kernel, linux-mm, netdev,
trond.myklebust
Cc: Peter Zijlstra
In-Reply-To: <20071214153907.770251000@chello.nl>
[-- Attachment #1: mm-reserve.patch --]
[-- Type: text/plain, Size: 14396 bytes --]
Generic reserve management code.
It provides methods to reserve and charge. Upon this, generic alloc/free style
reserve pools could be build, which could fully replace mempool_t
functionality.
It should also allow for a Banker's algorithm replacement of __GFP_NOFAIL.
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
---
include/linux/reserve.h | 54 +++++
mm/Makefile | 2
mm/reserve.c | 438 ++++++++++++++++++++++++++++++++++++++++++++++++
3 files changed, 493 insertions(+), 1 deletion(-)
Index: linux-2.6/include/linux/reserve.h
===================================================================
--- /dev/null
+++ linux-2.6/include/linux/reserve.h
@@ -0,0 +1,54 @@
+/*
+ * Memory reserve management.
+ *
+ * Copyright (C) 2007 Red Hat, Inc., Peter Zijlstra <pzijlstr@redhat.com>
+ *
+ * This file contains the public data structure and API definitions.
+ */
+
+#ifndef _LINUX_RESERVE_H
+#define _LINUX_RESERVE_H
+
+#include <linux/list.h>
+#include <linux/spinlock.h>
+
+struct mem_reserve {
+ struct mem_reserve *parent;
+ struct list_head children;
+ struct list_head siblings;
+
+ const char *name;
+
+ long pages;
+ long limit;
+ long usage;
+ spinlock_t lock; /* protects limit and usage */
+};
+
+extern struct mem_reserve mem_reserve_root;
+
+void mem_reserve_init(struct mem_reserve *res, const char *name,
+ struct mem_reserve *parent);
+int mem_reserve_connect(struct mem_reserve *new_child,
+ struct mem_reserve *node);
+int mem_reserve_disconnect(struct mem_reserve *node);
+
+int mem_reserve_pages_set(struct mem_reserve *res, long pages);
+int mem_reserve_pages_add(struct mem_reserve *res, long pages);
+int mem_reserve_pages_charge(struct mem_reserve *res, long pages,
+ int overcommit);
+
+int mem_reserve_kmalloc_set(struct mem_reserve *res, long bytes);
+int mem_reserve_kmalloc_charge(struct mem_reserve *res, long bytes,
+ int overcommit);
+
+struct kmem_cache;
+
+int mem_reserve_kmem_cache_set(struct mem_reserve *res,
+ struct kmem_cache *s,
+ int objects);
+int mem_reserve_kmem_cache_charge(struct mem_reserve *res,
+ long objs,
+ int overcommit);
+
+#endif /* _LINUX_RESERVE_H */
Index: linux-2.6/mm/Makefile
===================================================================
--- linux-2.6.orig/mm/Makefile
+++ linux-2.6/mm/Makefile
@@ -11,7 +11,7 @@ obj-y := bootmem.o filemap.o mempool.o
page_alloc.o page-writeback.o pdflush.o \
readahead.o swap.o truncate.o vmscan.o \
prio_tree.o util.o mmzone.o vmstat.o backing-dev.o \
- page_isolation.o $(mmu-y)
+ page_isolation.o reserve.o $(mmu-y)
obj-$(CONFIG_PROC_PAGE_MONITOR) += pagewalk.o
obj-$(CONFIG_BOUNCE) += bounce.o
Index: linux-2.6/mm/reserve.c
===================================================================
--- /dev/null
+++ linux-2.6/mm/reserve.c
@@ -0,0 +1,438 @@
+/*
+ * Memory reserve management.
+ *
+ * Copyright (C) 2007, Red Hat, Inc., Peter Zijlstra <pzijlstr@redhat.com>
+ *
+ * Description:
+ *
+ * Manage a set of memory reserves.
+ *
+ * A memory reserve is a reserve for a specified number of object of specified
+ * size. Since memory is managed in pages, this reserve demand is then
+ * translated into a page unit.
+ *
+ * So each reserve has a specified object limit, an object usage count and a
+ * number of pages required to back these objects.
+ *
+ * Usage is charged against a reserve, if the charge fails, the resource must
+ * not be allocated/used.
+ *
+ * The reserves are managed in a tree, and the resource demands (pages and
+ * limit) are propagated up the tree. Obviously the object limit will be
+ * meaningless as soon as the unit starts mixing, but the required page reserve
+ * (being of one unit) is still valid at the root.
+ *
+ * It is the page demand of the root node that is used to set the global
+ * reserve (adjust_memalloc_reserve() which sets zone->pages_emerg).
+ *
+ * As long as a subtree has the same usage unit, an aggregate node can be used
+ * to charge against, instead of the leaf nodes. However, do be consistent with
+ * who is charged, resource usage is not propagated up the tree (for
+ * performance reasons).
+ */
+
+#include <linux/reserve.h>
+#include <linux/mutex.h>
+#include <linux/mmzone.h>
+#include <linux/log2.h>
+#include <linux/proc_fs.h>
+#include <linux/seq_file.h>
+#include <linux/module.h>
+#include <linux/slab.h>
+
+static DEFINE_MUTEX(mem_reserve_mutex);
+
+/**
+ * @mem_reserve_root - the global reserve root
+ *
+ * The global reserve is empty, and has no limit unit, it merely
+ * acts as an aggregation point for reserves and an interface to
+ * adjust_memalloc_reserve().
+ */
+struct mem_reserve mem_reserve_root = {
+ .children = LIST_HEAD_INIT(mem_reserve_root.children),
+ .siblings = LIST_HEAD_INIT(mem_reserve_root.siblings),
+ .name = "total reserve",
+ .lock = __SPIN_LOCK_UNLOCKED(mem_reserve_root.lock),
+};
+
+EXPORT_SYMBOL_GPL(mem_reserve_root);
+
+/**
+ * mem_reserve_init - initialize a memory reserve object
+ * @res - the new reserve object
+ * @name - a name for this reserve
+ */
+void mem_reserve_init(struct mem_reserve *res, const char *name,
+ struct mem_reserve *parent)
+{
+ memset(res, 0, sizeof(*res));
+ INIT_LIST_HEAD(&res->children);
+ INIT_LIST_HEAD(&res->siblings);
+ res->name = name;
+ spin_lock_init(&res->lock);
+
+ if (parent)
+ mem_reserve_connect(res, parent);
+}
+
+EXPORT_SYMBOL_GPL(mem_reserve_init);
+
+/*
+ * propagate the pages and limit changes up the tree.
+ */
+static void __calc_reserve(struct mem_reserve *res, long pages, long limit)
+{
+ unsigned long flags;
+
+ for ( ; res; res = res->parent) {
+ res->pages += pages;
+
+ if (limit) {
+ spin_lock_irqsave(&res->lock, flags);
+ res->limit += limit;
+ spin_unlock_irqrestore(&res->lock, flags);
+ }
+ }
+}
+
+/**
+ * __mem_reserve_add - primitive to change the size of a reserve
+ * @res - reserve to change
+ * @pages - page delta
+ * @limit - usage limit delta
+ *
+ * Returns -ENOMEM when a size increase is not possible atm.
+ */
+static int __mem_reserve_add(struct mem_reserve *res, long pages, long limit)
+{
+ int ret = 0;
+ long reserve;
+
+ reserve = mem_reserve_root.pages;
+ __calc_reserve(res, pages, 0);
+ reserve = mem_reserve_root.pages - reserve;
+
+ if (reserve) {
+ ret = adjust_memalloc_reserve(reserve);
+ if (ret)
+ __calc_reserve(res, -pages, 0);
+ }
+
+ if (!ret)
+ __calc_reserve(res, 0, limit);
+
+ return ret;
+}
+
+/**
+ * __mem_reserve_charge - primitive to charge object usage to a reserve
+ * @res - reserve to charge
+ * @charge - size of the charge
+ * @overcommit - allow despite of limit (use with caution!)
+ *
+ * Returns non-zero on success, zero on failure.
+ */
+static
+int __mem_reserve_charge(struct mem_reserve *res, long charge, int overcommit)
+{
+ unsigned long flags;
+ int ret = 0;
+
+ spin_lock_irqsave(&res->lock, flags);
+ if (charge < 0 || res->usage + charge < res->limit || overcommit) {
+ res->usage += charge;
+ if (unlikely(res->usage < 0))
+ res->usage = 0;
+ ret = 1;
+ }
+ spin_unlock_irqrestore(&res->lock, flags);
+
+ return ret;
+}
+
+/**
+ * mem_reserve_connect - connect a reserve to another in a child-parent relation
+ * @new_child - the reserve node to connect (child)
+ * @node - the reserve node to connect to (parent)
+ *
+ * Returns -ENOMEM when the new connection would increase the reserve (parent
+ * is connected to mem_reserve_root) and there is no memory to do so.
+ *
+ * The child is _NOT_ connected on error.
+ */
+int mem_reserve_connect(struct mem_reserve *new_child, struct mem_reserve *node)
+{
+ int ret;
+
+ WARN_ON(!new_child->name);
+
+ mutex_lock(&mem_reserve_mutex);
+ new_child->parent = node;
+ list_add(&new_child->siblings, &node->children);
+ ret = __mem_reserve_add(node, new_child->pages, new_child->limit);
+ if (ret) {
+ new_child->parent = NULL;
+ list_del_init(&new_child->siblings);
+ }
+ mutex_unlock(&mem_reserve_mutex);
+
+ return ret;
+}
+
+EXPORT_SYMBOL_GPL(mem_reserve_connect);
+
+/**
+ * mem_reserve_disconnect - sever a nodes connection to the reserve tree
+ * @node - the node to disconnect
+ *
+ * Could, in theory, return -ENOMEM, but since disconnecting a node _should_
+ * only decrease the reserves that _should_ not happen.
+ */
+int mem_reserve_disconnect(struct mem_reserve *node)
+{
+ int ret;
+
+ BUG_ON(!node->parent);
+
+ mutex_lock(&mem_reserve_mutex);
+ ret = __mem_reserve_add(node->parent, -node->pages, -node->limit);
+ if (!ret) {
+ node->parent = NULL;
+ list_del_init(&node->siblings);
+ }
+ mutex_unlock(&mem_reserve_mutex);
+
+ return ret;
+}
+
+EXPORT_SYMBOL_GPL(mem_reserve_disconnect);
+
+#ifdef CONFIG_PROC_FS
+
+/*
+ * Simple output of the reserve tree in: /proc/reserve_info
+ * Example:
+ *
+ * localhost ~ # cat /proc/reserve_info
+ * total reserve 8156K (0/544817)
+ * total network reserve 8156K (0/544817)
+ * network TX reserve 196K (0/49)
+ * protocol TX pages 196K (0/49)
+ * network RX reserve 7960K (0/544768)
+ * IPv6 route cache 1372K (0/4096)
+ * IPv4 route cache 5468K (0/16384)
+ * SKB data reserve 1120K (0/524288)
+ * IPv6 fragment cache 560K (0/262144)
+ * IPv4 fragment cache 560K (0/262144)
+ */
+
+static void mem_reserve_show_item(struct seq_file *m, struct mem_reserve *res,
+ int nesting)
+{
+ int i;
+ struct mem_reserve *child;
+
+ for (i = 0; i < nesting; i++)
+ seq_puts(m, " ");
+
+ seq_printf(m, "%-30s %ldK (%ld/%ld)\n",
+ res->name, res->pages << (PAGE_SHIFT - 10),
+ res->usage, res->limit);
+
+ list_for_each_entry(child, &res->children, siblings)
+ mem_reserve_show_item(m, child, nesting+1);
+}
+
+static int mem_reserve_show(struct seq_file *m, void *v)
+{
+ mutex_lock(&mem_reserve_mutex);
+ mem_reserve_show_item(m, &mem_reserve_root, 0);
+ mutex_unlock(&mem_reserve_mutex);
+
+ return 0;
+}
+
+static int mem_reserve_open(struct inode *inode, struct file *file)
+{
+ return single_open(file, mem_reserve_show, NULL);
+}
+
+static const struct file_operations mem_reserve_opterations = {
+ .open = mem_reserve_open,
+ .read = seq_read,
+ .llseek = seq_lseek,
+ .release = single_release,
+};
+
+static __init int mem_reserve_proc_init(void)
+{
+ struct proc_dir_entry *entry;
+
+ entry = create_proc_entry("reserve_info", S_IRUSR, NULL);
+ if (entry)
+ entry->proc_fops = &mem_reserve_opterations;
+
+ return 0;
+}
+
+__initcall(mem_reserve_proc_init);
+
+#endif
+
+/*
+ * alloc_page helpers
+ */
+
+/**
+ * mem_reserve_pages_set - set reserves size in pages
+ * @res - reserve to set
+ * @pages - size in pages to set it to
+ *
+ * Returns -ENOMEM when it fails to set the reserve. On failure the old size
+ * is preserved.
+ */
+int mem_reserve_pages_set(struct mem_reserve *res, long pages)
+{
+ int ret;
+
+ mutex_lock(&mem_reserve_mutex);
+ pages -= res->pages;
+ ret = __mem_reserve_add(res, pages, pages);
+ mutex_unlock(&mem_reserve_mutex);
+
+ return ret;
+}
+
+EXPORT_SYMBOL_GPL(mem_reserve_pages_set);
+
+/**
+ * mem_reserve_pages_add - change the size in a relative way
+ * @res - reserve to change
+ * @pages - number of pages to add (or subtract when negative)
+ *
+ * Similar to mem_reserve_pages_set, except that the argument is relative instead
+ * of absolute.
+ *
+ * Returns -ENOMEM when it fails to increase.
+ */
+int mem_reserve_pages_add(struct mem_reserve *res, long pages)
+{
+ int ret;
+
+ mutex_lock(&mem_reserve_mutex);
+ ret = __mem_reserve_add(res, pages, pages);
+ mutex_unlock(&mem_reserve_mutex);
+
+ return ret;
+}
+
+/**
+ * mem_reserve_pages_charge - charge page usage to a reserve
+ * @res - reserve to charge
+ * @pages - size to charge
+ * @overcommit - disregard the usage limit (use with caution!)
+ *
+ * Returns non-zero on success.
+ */
+int mem_reserve_pages_charge(struct mem_reserve *res, long pages, int overcommit)
+{
+ return __mem_reserve_charge(res, pages, overcommit);
+}
+
+EXPORT_SYMBOL_GPL(mem_reserve_pages_charge);
+
+/*
+ * kmalloc helpers
+ */
+
+/**
+ * mem_reserve_kmalloc_set - set this reserve to bytes worth of kmalloc
+ * @res - reserve to change
+ * @bytes - size in bytes to reserve
+ *
+ * Returns -ENOMEM on failure.
+ */
+int mem_reserve_kmalloc_set(struct mem_reserve *res, long bytes)
+{
+ int ret;
+ long pages;
+
+ mutex_lock(&mem_reserve_mutex);
+ pages = kestimate(GFP_ATOMIC, bytes);
+ pages -= res->pages;
+ bytes -= res->limit;
+ ret = __mem_reserve_add(res, pages, bytes);
+ mutex_unlock(&mem_reserve_mutex);
+
+ return ret;
+}
+
+EXPORT_SYMBOL_GPL(mem_reserve_kmalloc_set);
+
+/**
+ * mem_reserve_kmalloc_charge - charge bytes to a reserve
+ * @res - reserve to charge
+ * @bytes - bytes to charge
+ * @overcommit - disregard the usage limit (use with caution!)
+ *
+ * Returns non-zero on success.
+ */
+int mem_reserve_kmalloc_charge(struct mem_reserve *res, long bytes,
+ int overcommit)
+{
+ if (bytes < 0)
+ bytes = -roundup_pow_of_two(-bytes);
+ else
+ bytes = roundup_pow_of_two(bytes);
+
+ return __mem_reserve_charge(res, bytes, overcommit);
+}
+
+EXPORT_SYMBOL_GPL(mem_reserve_kmalloc_charge);
+
+/*
+ * kmem_cache helpers
+ */
+
+/**
+ * mem_reserve_kmem_cache_set - set reserve to @objects worth of kmem_cache_alloc of @s
+ * @res - reserve to set
+ * @s - kmem_cache to reserve from
+ * @objects - number of objects to reserve
+ *
+ * Returns -ENOMEM on failure.
+ */
+int mem_reserve_kmem_cache_set(struct mem_reserve *res, struct kmem_cache *s,
+ int objects)
+{
+ int ret;
+ long pages;
+
+ mutex_lock(&mem_reserve_mutex);
+ pages = kmem_estimate_pages(s, GFP_ATOMIC, objects);
+ pages -= res->pages;
+ objects -= res->limit;
+ ret = __mem_reserve_add(res, pages, objects);
+ mutex_unlock(&mem_reserve_mutex);
+
+ return ret;
+}
+
+EXPORT_SYMBOL_GPL(mem_reserve_kmem_cache_set);
+
+/**
+ * mem_reserve_kmem_cache_charge - charge (or uncharge) usage of objs
+ * @res - reserve to charge
+ * @objs - objects to charge for
+ * @overcommit - disregard the usage limit (use with caution!)
+ *
+ * Returns non-zero on success.
+ */
+int mem_reserve_kmem_cache_charge(struct mem_reserve *res, long objs,
+ int overcommit)
+{
+ return __mem_reserve_charge(res, objs, overcommit);
+}
+
+EXPORT_SYMBOL_GPL(mem_reserve_kmem_cache_charge);
--
^ permalink raw reply
* [PATCH 26/29] nfs: teach the NFS client how to treat PG_swapcache pages
From: Peter Zijlstra @ 2007-12-14 15:39 UTC (permalink / raw)
To: Linus Torvalds, Andrew Morton, linux-kernel, linux-mm, netdev,
trond.myklebust
Cc: Peter Zijlstra
In-Reply-To: <20071214153907.770251000@chello.nl>
[-- Attachment #1: nfs-swapcache.patch --]
[-- Type: text/plain, Size: 12547 bytes --]
Replace all relevant occurences of page->index and page->mapping in the NFS
client with the new page_file_index() and page_file_mapping() functions.
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
---
fs/nfs/file.c | 8 ++++----
fs/nfs/internal.h | 7 ++++---
fs/nfs/pagelist.c | 6 +++---
fs/nfs/read.c | 6 +++---
fs/nfs/write.c | 49 +++++++++++++++++++++++++------------------------
5 files changed, 39 insertions(+), 37 deletions(-)
Index: linux-2.6/fs/nfs/file.c
===================================================================
--- linux-2.6.orig/fs/nfs/file.c
+++ linux-2.6/fs/nfs/file.c
@@ -357,7 +357,7 @@ static void nfs_invalidate_page(struct p
if (offset != 0)
return;
/* Cancel any unstarted writes on this page */
- nfs_wb_page_cancel(page->mapping->host, page);
+ nfs_wb_page_cancel(page_file_mapping(page)->host, page);
}
static int nfs_release_page(struct page *page, gfp_t gfp)
@@ -368,7 +368,7 @@ static int nfs_release_page(struct page
static int nfs_launder_page(struct page *page)
{
- return nfs_wb_page(page->mapping->host, page);
+ return nfs_wb_page(page_file_mapping(page)->host, page);
}
const struct address_space_operations nfs_file_aops = {
@@ -397,13 +397,13 @@ static int nfs_vm_page_mkwrite(struct vm
loff_t offset;
lock_page(page);
- mapping = page->mapping;
+ mapping = page_file_mapping(page);
if (mapping != vma->vm_file->f_path.dentry->d_inode->i_mapping) {
unlock_page(page);
return -EINVAL;
}
pagelen = nfs_page_length(page);
- offset = (loff_t)page->index << PAGE_CACHE_SHIFT;
+ offset = (loff_t)page_file_index(page) << PAGE_CACHE_SHIFT;
unlock_page(page);
/*
Index: linux-2.6/fs/nfs/pagelist.c
===================================================================
--- linux-2.6.orig/fs/nfs/pagelist.c
+++ linux-2.6/fs/nfs/pagelist.c
@@ -77,11 +77,11 @@ nfs_create_request(struct nfs_open_conte
* update_nfs_request below if the region is not locked. */
req->wb_page = page;
atomic_set(&req->wb_complete, 0);
- req->wb_index = page->index;
+ req->wb_index = page_file_index(page);
page_cache_get(page);
BUG_ON(PagePrivate(page));
BUG_ON(!PageLocked(page));
- BUG_ON(page->mapping->host != inode);
+ BUG_ON(page_file_mapping(page)->host != inode);
req->wb_offset = offset;
req->wb_pgbase = offset;
req->wb_bytes = count;
@@ -383,7 +383,7 @@ void nfs_pageio_cond_complete(struct nfs
* nfs_scan_list - Scan a list for matching requests
* @nfsi: NFS inode
* @dst: Destination list
- * @idx_start: lower bound of page->index to scan
+ * @idx_start: lower bound of page_file_index(page) to scan
* @npages: idx_start + npages sets the upper bound to scan.
* @tag: tag to scan for
*
Index: linux-2.6/fs/nfs/read.c
===================================================================
--- linux-2.6.orig/fs/nfs/read.c
+++ linux-2.6/fs/nfs/read.c
@@ -460,11 +460,11 @@ static const struct rpc_call_ops nfs_rea
int nfs_readpage(struct file *file, struct page *page)
{
struct nfs_open_context *ctx;
- struct inode *inode = page->mapping->host;
+ struct inode *inode = page_file_mapping(page)->host;
int error;
dprintk("NFS: nfs_readpage (%p %ld@%lu)\n",
- page, PAGE_CACHE_SIZE, page->index);
+ page, PAGE_CACHE_SIZE, page_file_index(page));
nfs_inc_stats(inode, NFSIOS_VFSREADPAGE);
nfs_add_stats(inode, NFSIOS_READPAGES, 1);
@@ -511,7 +511,7 @@ static int
readpage_async_filler(void *data, struct page *page)
{
struct nfs_readdesc *desc = (struct nfs_readdesc *)data;
- struct inode *inode = page->mapping->host;
+ struct inode *inode = page_file_mapping(page)->host;
struct nfs_page *new;
unsigned int len;
int error;
Index: linux-2.6/fs/nfs/write.c
===================================================================
--- linux-2.6.orig/fs/nfs/write.c
+++ linux-2.6/fs/nfs/write.c
@@ -126,7 +126,7 @@ static struct nfs_page *nfs_page_find_re
static struct nfs_page *nfs_page_find_request(struct page *page)
{
- struct inode *inode = page->mapping->host;
+ struct inode *inode = page_file_mapping(page)->host;
struct nfs_page *req = NULL;
spin_lock(&inode->i_lock);
@@ -138,13 +138,13 @@ static struct nfs_page *nfs_page_find_re
/* Adjust the file length if we're writing beyond the end */
static void nfs_grow_file(struct page *page, unsigned int offset, unsigned int count)
{
- struct inode *inode = page->mapping->host;
+ struct inode *inode = page_file_mapping(page)->host;
loff_t end, i_size = i_size_read(inode);
pgoff_t end_index = (i_size - 1) >> PAGE_CACHE_SHIFT;
- if (i_size > 0 && page->index < end_index)
+ if (i_size > 0 && page_file_index(page) < end_index)
return;
- end = ((loff_t)page->index << PAGE_CACHE_SHIFT) + ((loff_t)offset+count);
+ end = page_offset(page) + ((loff_t)offset+count);
if (i_size >= end)
return;
nfs_inc_stats(inode, NFSIOS_EXTENDWRITE);
@@ -155,7 +155,7 @@ static void nfs_grow_file(struct page *p
static void nfs_set_pageerror(struct page *page)
{
SetPageError(page);
- nfs_zap_mapping(page->mapping->host, page->mapping);
+ nfs_zap_mapping(page_file_mapping(page)->host, page_file_mapping(page));
}
/* We can set the PG_uptodate flag if we see that a write request
@@ -187,7 +187,7 @@ static int nfs_writepage_setup(struct nf
ret = PTR_ERR(req);
if (ret != -EBUSY)
return ret;
- ret = nfs_wb_page(page->mapping->host, page);
+ ret = nfs_wb_page(page_file_mapping(page)->host, page);
if (ret != 0)
return ret;
}
@@ -221,7 +221,7 @@ static int nfs_set_page_writeback(struct
int ret = test_set_page_writeback(page);
if (!ret) {
- struct inode *inode = page->mapping->host;
+ struct inode *inode = page_file_mapping(page)->host;
struct nfs_server *nfss = NFS_SERVER(inode);
if (atomic_long_inc_return(&nfss->writeback) >
@@ -233,7 +233,7 @@ static int nfs_set_page_writeback(struct
static void nfs_end_page_writeback(struct page *page)
{
- struct inode *inode = page->mapping->host;
+ struct inode *inode = page_file_mapping(page)->host;
struct nfs_server *nfss = NFS_SERVER(inode);
end_page_writeback(page);
@@ -248,7 +248,7 @@ static void nfs_end_page_writeback(struc
static int nfs_page_async_flush(struct nfs_pageio_descriptor *pgio,
struct page *page)
{
- struct inode *inode = page->mapping->host;
+ struct inode *inode = page_file_mapping(page)->host;
struct nfs_inode *nfsi = NFS_I(inode);
struct nfs_page *req;
int ret;
@@ -294,7 +294,7 @@ static int nfs_page_async_flush(struct n
static int nfs_do_writepage(struct page *page, struct writeback_control *wbc, struct nfs_pageio_descriptor *pgio)
{
- struct inode *inode = page->mapping->host;
+ struct inode *inode = page_file_mapping(page)->host;
nfs_inc_stats(inode, NFSIOS_VFSWRITEPAGE);
nfs_add_stats(inode, NFSIOS_WRITEPAGES, 1);
@@ -311,7 +311,7 @@ static int nfs_writepage_locked(struct p
struct nfs_pageio_descriptor pgio;
int err;
- nfs_pageio_init_write(&pgio, page->mapping->host, wb_priority(wbc));
+ nfs_pageio_init_write(&pgio, page_file_mapping(page)->host, wb_priority(wbc));
err = nfs_do_writepage(page, wbc, &pgio);
nfs_pageio_complete(&pgio);
if (err < 0)
@@ -442,7 +442,8 @@ nfs_mark_request_commit(struct nfs_page
NFS_PAGE_TAG_COMMIT);
spin_unlock(&inode->i_lock);
inc_zone_page_state(req->wb_page, NR_UNSTABLE_NFS);
- inc_bdi_stat(req->wb_page->mapping->backing_dev_info, BDI_RECLAIMABLE);
+ inc_bdi_stat(page_file_mapping(req->wb_page)->backing_dev_info,
+ BDI_RECLAIMABLE);
__mark_inode_dirty(inode, I_DIRTY_DATASYNC);
}
@@ -529,7 +530,7 @@ static void nfs_cancel_commit_list(struc
while(!list_empty(head)) {
req = nfs_list_entry(head->next);
dec_zone_page_state(req->wb_page, NR_UNSTABLE_NFS);
- dec_bdi_stat(req->wb_page->mapping->backing_dev_info,
+ dec_bdi_stat(page_file_mapping(req->wb_page)->backing_dev_info,
BDI_RECLAIMABLE);
nfs_list_remove_request(req);
clear_bit(PG_NEED_COMMIT, &(req)->wb_flags);
@@ -543,7 +544,7 @@ static void nfs_cancel_commit_list(struc
* nfs_scan_commit - Scan an inode for commit requests
* @inode: NFS inode to scan
* @dst: destination list
- * @idx_start: lower bound of page->index to scan.
+ * @idx_start: lower bound of page_file_index(page) to scan.
* @npages: idx_start + npages sets the upper bound to scan.
*
* Moves requests from the inode's 'commit' request list.
@@ -579,7 +580,7 @@ static inline int nfs_scan_commit(struct
static struct nfs_page * nfs_update_request(struct nfs_open_context* ctx,
struct page *page, unsigned int offset, unsigned int bytes)
{
- struct address_space *mapping = page->mapping;
+ struct address_space *mapping = page_file_mapping(page);
struct inode *inode = mapping->host;
struct nfs_page *req, *new = NULL;
pgoff_t rqend, end;
@@ -681,7 +682,7 @@ int nfs_flush_incompatible(struct file *
nfs_release_request(req);
if (!do_flush)
return 0;
- status = nfs_wb_page(page->mapping->host, page);
+ status = nfs_wb_page(page_file_mapping(page)->host, page);
} while (status == 0);
return status;
}
@@ -696,7 +697,7 @@ int nfs_updatepage(struct file *file, st
unsigned int offset, unsigned int count)
{
struct nfs_open_context *ctx = nfs_file_open_context(file);
- struct inode *inode = page->mapping->host;
+ struct inode *inode = page_file_mapping(page)->host;
int status = 0;
nfs_inc_stats(inode, NFSIOS_VFSUPDATEPAGE);
@@ -952,7 +953,7 @@ static void nfs_writeback_done_partial(s
}
if (nfs_write_need_commit(data)) {
- struct inode *inode = page->mapping->host;
+ struct inode *inode = page_file_mapping(page)->host;
spin_lock(&inode->i_lock);
if (test_bit(PG_NEED_RESCHED, &req->wb_flags)) {
@@ -1191,7 +1192,7 @@ nfs_commit_list(struct inode *inode, str
nfs_list_remove_request(req);
nfs_mark_request_commit(req);
dec_zone_page_state(req->wb_page, NR_UNSTABLE_NFS);
- dec_bdi_stat(req->wb_page->mapping->backing_dev_info,
+ dec_bdi_stat(page_file_mapping(req->wb_page)->backing_dev_info,
BDI_RECLAIMABLE);
nfs_clear_page_tag_locked(req);
}
@@ -1218,7 +1219,7 @@ static void nfs_commit_done(struct rpc_t
nfs_list_remove_request(req);
clear_bit(PG_NEED_COMMIT, &(req)->wb_flags);
dec_zone_page_state(req->wb_page, NR_UNSTABLE_NFS);
- dec_bdi_stat(req->wb_page->mapping->backing_dev_info,
+ dec_bdi_stat(page_file_mapping(req->wb_page)->backing_dev_info,
BDI_RECLAIMABLE);
dprintk("NFS: commit (%s/%Ld %d@%Ld)",
@@ -1384,7 +1385,7 @@ int nfs_wb_page_cancel(struct inode *ino
loff_t range_start = page_offset(page);
loff_t range_end = range_start + (loff_t)(PAGE_CACHE_SIZE - 1);
struct writeback_control wbc = {
- .bdi = page->mapping->backing_dev_info,
+ .bdi = page_file_mapping(page)->backing_dev_info,
.sync_mode = WB_SYNC_ALL,
.nr_to_write = LONG_MAX,
.range_start = range_start,
@@ -1417,7 +1418,7 @@ int nfs_wb_page_cancel(struct inode *ino
}
if (!PagePrivate(page))
return 0;
- ret = nfs_sync_mapping_wait(page->mapping, &wbc, FLUSH_INVALIDATE);
+ ret = nfs_sync_mapping_wait(page_file_mapping(page), &wbc, FLUSH_INVALIDATE);
out:
return ret;
}
@@ -1428,7 +1429,7 @@ static int nfs_wb_page_priority(struct i
loff_t range_start = page_offset(page);
loff_t range_end = range_start + (loff_t)(PAGE_CACHE_SIZE - 1);
struct writeback_control wbc = {
- .bdi = page->mapping->backing_dev_info,
+ .bdi = page_file_mapping(page)->backing_dev_info,
.sync_mode = WB_SYNC_ALL,
.nr_to_write = LONG_MAX,
.range_start = range_start,
@@ -1444,7 +1445,7 @@ static int nfs_wb_page_priority(struct i
}
if (!PagePrivate(page))
return 0;
- ret = nfs_sync_mapping_wait(page->mapping, &wbc, how);
+ ret = nfs_sync_mapping_wait(page_file_mapping(page), &wbc, how);
if (ret >= 0)
return 0;
out:
Index: linux-2.6/fs/nfs/internal.h
===================================================================
--- linux-2.6.orig/fs/nfs/internal.h
+++ linux-2.6/fs/nfs/internal.h
@@ -248,13 +248,14 @@ void nfs_super_set_maxbytes(struct super
static inline
unsigned int nfs_page_length(struct page *page)
{
- loff_t i_size = i_size_read(page->mapping->host);
+ loff_t i_size = i_size_read(page_file_mapping(page)->host);
if (i_size > 0) {
+ pgoff_t page_index = page_file_index(page);
pgoff_t end_index = (i_size - 1) >> PAGE_CACHE_SHIFT;
- if (page->index < end_index)
+ if (page_index < end_index)
return PAGE_CACHE_SIZE;
- if (page->index == end_index)
+ if (page_index == end_index)
return ((i_size - 1) & ~PAGE_CACHE_MASK) + 1;
}
return 0;
--
^ permalink raw reply
* [PATCH 29/29] nfs: fix various memory recursions possible with swap over NFS.
From: Peter Zijlstra @ 2007-12-14 15:39 UTC (permalink / raw)
To: Linus Torvalds, Andrew Morton, linux-kernel, linux-mm, netdev,
trond.myklebust
Cc: Peter Zijlstra
In-Reply-To: <20071214153907.770251000@chello.nl>
[-- Attachment #1: nfs-alloc-recursions.patch --]
[-- Type: text/plain, Size: 1861 bytes --]
GFP_NOFS is not enough, since swap traffic is IO, hence fall back to GFP_NOIO.
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
---
fs/nfs/pagelist.c | 2 +-
fs/nfs/write.c | 6 +++---
2 files changed, 4 insertions(+), 4 deletions(-)
Index: linux-2.6/fs/nfs/write.c
===================================================================
--- linux-2.6.orig/fs/nfs/write.c
+++ linux-2.6/fs/nfs/write.c
@@ -44,7 +44,7 @@ static struct kmem_cache *nfs_wdata_cach
struct nfs_write_data *nfs_commit_alloc(void)
{
- struct nfs_write_data *p = kmem_cache_alloc(nfs_wdata_cachep, GFP_NOFS);
+ struct nfs_write_data *p = kmem_cache_alloc(nfs_wdata_cachep, GFP_NOIO);
if (p) {
memset(p, 0, sizeof(*p));
@@ -68,7 +68,7 @@ void nfs_commit_free(struct nfs_write_da
struct nfs_write_data *nfs_writedata_alloc(unsigned int pagecount)
{
- struct nfs_write_data *p = kmem_cache_alloc(nfs_wdata_cachep, GFP_NOFS);
+ struct nfs_write_data *p = kmem_cache_alloc(nfs_wdata_cachep, GFP_NOIO);
if (p) {
memset(p, 0, sizeof(*p));
@@ -77,7 +77,7 @@ struct nfs_write_data *nfs_writedata_all
if (pagecount <= ARRAY_SIZE(p->page_array))
p->pagevec = p->page_array;
else {
- p->pagevec = kcalloc(pagecount, sizeof(struct page *), GFP_NOFS);
+ p->pagevec = kcalloc(pagecount, sizeof(struct page *), GFP_NOIO);
if (!p->pagevec) {
kmem_cache_free(nfs_wdata_cachep, p);
p = NULL;
Index: linux-2.6/fs/nfs/pagelist.c
===================================================================
--- linux-2.6.orig/fs/nfs/pagelist.c
+++ linux-2.6/fs/nfs/pagelist.c
@@ -27,7 +27,7 @@ static inline struct nfs_page *
nfs_page_alloc(void)
{
struct nfs_page *p;
- p = kmem_cache_alloc(nfs_page_cachep, GFP_KERNEL);
+ p = kmem_cache_alloc(nfs_page_cachep, GFP_NOIO);
if (p) {
memset(p, 0, sizeof(*p));
INIT_LIST_HEAD(&p->wb_list);
--
^ permalink raw reply
* [PATCH 18/29] netvm: filter emergency skbs.
From: Peter Zijlstra @ 2007-12-14 15:39 UTC (permalink / raw)
To: Linus Torvalds, Andrew Morton, linux-kernel, linux-mm, netdev,
trond.myklebust
Cc: Peter Zijlstra
In-Reply-To: <20071214153907.770251000@chello.nl>
[-- Attachment #1: netvm-sk_filter.patch --]
[-- Type: text/plain, Size: 765 bytes --]
Toss all emergency packets not for a SOCK_MEMALLOC socket. This ensures our
precious memory reserve doesn't get stuck waiting for user-space.
The correctness of this approach relies on the fact that networks must be
assumed lossy.
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
---
include/net/sock.h | 3 +++
1 file changed, 3 insertions(+)
Index: linux-2.6/include/net/sock.h
===================================================================
--- linux-2.6.orig/include/net/sock.h
+++ linux-2.6/include/net/sock.h
@@ -930,6 +930,9 @@ static inline int sk_filter(struct sock
{
int err;
struct sk_filter *filter;
+
+ if (skb_emergency(skb) && !sk_has_memalloc(sk))
+ return -ENOMEM;
err = security_sock_rcv_skb(sk, skb);
if (err)
--
^ permalink raw reply
* [PATCH 23/29] mm: add support for non block device backed swap files
From: Peter Zijlstra @ 2007-12-14 15:39 UTC (permalink / raw)
To: Linus Torvalds, Andrew Morton, linux-kernel, linux-mm, netdev,
trond.myklebust
Cc: Peter Zijlstra
In-Reply-To: <20071214153907.770251000@chello.nl>
[-- Attachment #1: mm-swapfile.patch --]
[-- Type: text/plain, Size: 11689 bytes --]
New addres_space_operations methods are added:
int swapfile(struct address_space *, int);
int swap_out(struct file *, struct page *, struct writeback_control *);
int swap_in(struct file *, struct page *);
When during sys_swapon() the swapfile() method is found and returns no error
the swapper_space.a_ops will proxy to sis->swap_file->f_mapping->a_ops, and
make use of swap_{out,in}() to write/read swapcache pages.
The swapfile method will be used to communicate to the address_space that the
VM relies on it, and the address_space should take adequate measures (like
reserving memory for mempools or the like).
This new interface can be used to obviate the need for ->bmap in the swapfile
code. A filesystem would need to load (and maybe even allocate) the full block
map for a file into memory and pin it there on ->swapfile(,1) so that
->swap_{out,in}() have instant access to it. It can be released on
->swapfile(,0).
The reason to provide ->swap_{out,in}() over using {write,read}page() is to
1) make a distinction between swapcache and pagecache pages, and
2) to provide a struct file * for credential context (normally not needed
in the context of writepage, as the page content is normally dirtied
using either of the following interfaces:
write_{begin,end}()
{prepare,commit}_write()
page_mkwrite()
which do have the file context.
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
---
Documentation/filesystems/Locking | 19 ++++++++++++
Documentation/filesystems/vfs.txt | 17 +++++++++++
include/linux/buffer_head.h | 2 -
include/linux/fs.h | 8 +++++
include/linux/swap.h | 3 +
mm/Kconfig | 3 +
mm/page_io.c | 58 ++++++++++++++++++++++++++++++++++++++
mm/swap_state.c | 5 +++
mm/swapfile.c | 22 +++++++++++++-
9 files changed, 135 insertions(+), 2 deletions(-)
Index: linux-2.6/include/linux/swap.h
===================================================================
--- linux-2.6.orig/include/linux/swap.h
+++ linux-2.6/include/linux/swap.h
@@ -164,6 +164,7 @@ enum {
SWP_USED = (1 << 0), /* is slot in swap_info[] used? */
SWP_WRITEOK = (1 << 1), /* ok to write to this swap? */
SWP_ACTIVE = (SWP_USED | SWP_WRITEOK),
+ SWP_FILE = (1 << 2), /* file swap area */
/* add others here before... */
SWP_SCANNING = (1 << 8), /* refcount in scan_swap_map */
};
@@ -261,6 +262,8 @@ extern void swap_unplug_io_fn(struct bac
/* linux/mm/page_io.c */
extern int swap_readpage(struct file *, struct page *);
extern int swap_writepage(struct page *page, struct writeback_control *wbc);
+extern void swap_sync_page(struct page *page);
+extern int swap_set_page_dirty(struct page *page);
extern void end_swap_bio_read(struct bio *bio, int err);
/* linux/mm/swap_state.c */
Index: linux-2.6/mm/page_io.c
===================================================================
--- linux-2.6.orig/mm/page_io.c
+++ linux-2.6/mm/page_io.c
@@ -17,6 +17,7 @@
#include <linux/bio.h>
#include <linux/swapops.h>
#include <linux/writeback.h>
+#include <linux/buffer_head.h>
#include <asm/pgtable.h>
static struct bio *get_swap_bio(gfp_t gfp_flags, pgoff_t index,
@@ -102,6 +103,18 @@ int swap_writepage(struct page *page, st
unlock_page(page);
goto out;
}
+#ifdef CONFIG_SWAP_FILE
+ {
+ struct swap_info_struct *sis = page_swap_info(page);
+ if (sis->flags & SWP_FILE) {
+ ret = sis->swap_file->f_mapping->
+ a_ops->swap_out(sis->swap_file, page, wbc);
+ if (!ret)
+ count_vm_event(PSWPOUT);
+ return ret;
+ }
+ }
+#endif
bio = get_swap_bio(GFP_NOIO, page_private(page), page,
end_swap_bio_write);
if (bio == NULL) {
@@ -120,6 +133,39 @@ out:
return ret;
}
+#ifdef CONFIG_SWAP_FILE
+void swap_sync_page(struct page *page)
+{
+ struct swap_info_struct *sis = page_swap_info(page);
+
+ if (sis->flags & SWP_FILE) {
+ const struct address_space_operations * a_ops =
+ sis->swap_file->f_mapping->a_ops;
+ if (a_ops->sync_page)
+ a_ops->sync_page(page);
+ } else
+ block_sync_page(page);
+}
+
+int swap_set_page_dirty(struct page *page)
+{
+ struct swap_info_struct *sis = page_swap_info(page);
+
+ if (sis->flags & SWP_FILE) {
+ const struct address_space_operations * a_ops =
+ sis->swap_file->f_mapping->a_ops;
+ int (*spd)(struct page *) = a_ops->set_page_dirty;
+#ifdef CONFIG_BLOCK
+ if (!spd)
+ spd = __set_page_dirty_buffers;
+#endif
+ return (*spd)(page);
+ }
+
+ return __set_page_dirty_nobuffers(page);
+}
+#endif
+
int swap_readpage(struct file *file, struct page *page)
{
struct bio *bio;
@@ -127,6 +173,18 @@ int swap_readpage(struct file *file, str
BUG_ON(!PageLocked(page));
ClearPageUptodate(page);
+#ifdef CONFIG_SWAP_FILE
+ {
+ struct swap_info_struct *sis = page_swap_info(page);
+ if (sis->flags & SWP_FILE) {
+ ret = sis->swap_file->f_mapping->
+ a_ops->swap_in(sis->swap_file, page);
+ if (!ret)
+ count_vm_event(PSWPIN);
+ return ret;
+ }
+ }
+#endif
bio = get_swap_bio(GFP_KERNEL, page_private(page), page,
end_swap_bio_read);
if (bio == NULL) {
Index: linux-2.6/mm/swap_state.c
===================================================================
--- linux-2.6.orig/mm/swap_state.c
+++ linux-2.6/mm/swap_state.c
@@ -27,8 +27,13 @@
*/
static const struct address_space_operations swap_aops = {
.writepage = swap_writepage,
+#ifdef CONFIG_SWAP_FILE
+ .sync_page = swap_sync_page,
+ .set_page_dirty = swap_set_page_dirty,
+#else
.sync_page = block_sync_page,
.set_page_dirty = __set_page_dirty_nobuffers,
+#endif
.migratepage = migrate_page,
};
Index: linux-2.6/mm/swapfile.c
===================================================================
--- linux-2.6.orig/mm/swapfile.c
+++ linux-2.6/mm/swapfile.c
@@ -1015,6 +1015,13 @@ static void destroy_swap_extents(struct
list_del(&se->list);
kfree(se);
}
+#ifdef CONFIG_SWAP_FILE
+ if (sis->flags & SWP_FILE) {
+ sis->flags &= ~SWP_FILE;
+ sis->swap_file->f_mapping->a_ops->
+ swapfile(sis->swap_file->f_mapping, 0);
+ }
+#endif
}
/*
@@ -1107,6 +1114,19 @@ static int setup_swap_extents(struct swa
goto done;
}
+#ifdef CONFIG_SWAP_FILE
+ if (sis->swap_file->f_mapping->a_ops->swapfile) {
+ ret = sis->swap_file->f_mapping->a_ops->
+ swapfile(sis->swap_file->f_mapping, 1);
+ if (!ret) {
+ sis->flags |= SWP_FILE;
+ ret = add_swap_extent(sis, 0, sis->max, 0);
+ *span = sis->pages;
+ }
+ goto done;
+ }
+#endif
+
blkbits = inode->i_blkbits;
blocks_per_page = PAGE_SIZE >> blkbits;
@@ -1671,7 +1691,7 @@ asmlinkage long sys_swapon(const char __
mutex_lock(&swapon_mutex);
spin_lock(&swap_lock);
- p->flags = SWP_ACTIVE;
+ p->flags |= SWP_WRITEOK;
nr_swap_pages += nr_good_pages;
total_swap_pages += nr_good_pages;
Index: linux-2.6/include/linux/fs.h
===================================================================
--- linux-2.6.orig/include/linux/fs.h
+++ linux-2.6/include/linux/fs.h
@@ -479,6 +479,14 @@ struct address_space_operations {
int (*migratepage) (struct address_space *,
struct page *, struct page *);
int (*launder_page) (struct page *);
+
+ /*
+ * swapfile support
+ */
+ int (*swapfile)(struct address_space *, int);
+ int (*swap_out)(struct file *file, struct page *page,
+ struct writeback_control *wbc);
+ int (*swap_in)(struct file *file, struct page *page);
};
/*
Index: linux-2.6/Documentation/filesystems/Locking
===================================================================
--- linux-2.6.orig/Documentation/filesystems/Locking
+++ linux-2.6/Documentation/filesystems/Locking
@@ -171,6 +171,9 @@ prototypes:
int (*direct_IO)(int, struct kiocb *, const struct iovec *iov,
loff_t offset, unsigned long nr_segs);
int (*launder_page) (struct page *);
+ int (*swapfile) (struct address_space *, int);
+ int (*swap_out) (struct file *, struct page *, struct writeback_control *);
+ int (*swap_in) (struct file *, struct page *);
locking rules:
All except set_page_dirty may block
@@ -192,6 +195,9 @@ invalidatepage: no yes
releasepage: no yes
direct_IO: no
launder_page: no yes
+swapfile no
+swap_out no yes, unlocks
+swap_in no yes, unlocks
->prepare_write(), ->commit_write(), ->sync_page() and ->readpage()
may be called from the request handler (/dev/loop).
@@ -291,6 +297,19 @@ cleaned, or an error value if not. Note
getting mapped back in and redirtied, it needs to be kept locked
across the entire operation.
+ ->swapfile() will be called with a non zero argument on address spaces
+backing non block device backed swapfiles. A return value of zero indicates
+success. In which case this address space can be used for backing swapspace.
+The swapspace operations will be proxied to the address space operations.
+Swapoff will call this method with a zero argument to release the address
+space.
+
+ ->swap_out() when swapfile() returned success, this method is used to
+write the swap page.
+
+ ->swap_in() when swapfile() returned success, this method is used to
+read the swap page.
+
Note: currently almost all instances of address_space methods are
using BKL for internal serialization and that's one of the worst sources
of contention. Normally they are calling library functions (in fs/buffer.c)
Index: linux-2.6/mm/Kconfig
===================================================================
--- linux-2.6.orig/mm/Kconfig
+++ linux-2.6/mm/Kconfig
@@ -185,6 +185,9 @@ config BOUNCE
def_bool y
depends on BLOCK && MMU && (ZONE_DMA || HIGHMEM)
+config SWAP_FILE
+ def_bool n
+
config NR_QUICK
int
depends on QUICKLIST
Index: linux-2.6/include/linux/buffer_head.h
===================================================================
--- linux-2.6.orig/include/linux/buffer_head.h
+++ linux-2.6/include/linux/buffer_head.h
@@ -335,7 +335,7 @@ static inline void invalidate_inode_buff
static inline int remove_inode_buffers(struct inode *inode) { return 1; }
static inline int sync_mapping_buffers(struct address_space *mapping) { return 0; }
static inline void invalidate_bdev(struct block_device *bdev) {}
-
+static inline void block_sync_page(struct page *) { }
#endif /* CONFIG_BLOCK */
#endif /* _LINUX_BUFFER_HEAD_H */
Index: linux-2.6/Documentation/filesystems/vfs.txt
===================================================================
--- linux-2.6.orig/Documentation/filesystems/vfs.txt
+++ linux-2.6/Documentation/filesystems/vfs.txt
@@ -542,6 +542,10 @@ struct address_space_operations {
/* migrate the contents of a page to the specified target */
int (*migratepage) (struct page *, struct page *);
int (*launder_page) (struct page *);
+ int (*swapfile)(struct address_space *, int);
+ int (*swap_out)(struct file *file, struct page *page,
+ struct writeback_control *wbc);
+ int (*swap_in)(struct file *file, struct page *page);
};
writepage: called by the VM to write a dirty page to backing store.
@@ -727,6 +731,19 @@ struct address_space_operations {
prevent redirtying the page, it is kept locked during the whole
operation.
+ swapfile: Called with a non-zero argument when swapon is used on a file. A
+ return value of zero indicates success. In which case this
+ address_space can be used to back swapspace. The swapspace operations
+ will be proxied to this address space's ->swap_{out,in} methods.
+ Swapoff will call this method with a zero argument to release the
+ address space.
+
+ swap_out: Called to write a swapcache page to a backing store, similar to
+ writepage.
+
+ swap_in: Called to read a swapcache page from a backing store, similar to
+ readpage.
+
The File Object
===============
--
^ permalink raw reply
* [PATCH 15/29] netvm: network reserve infrastructure
From: Peter Zijlstra @ 2007-12-14 15:39 UTC (permalink / raw)
To: Linus Torvalds, Andrew Morton, linux-kernel, linux-mm, netdev,
trond.myklebust
Cc: Peter Zijlstra
In-Reply-To: <20071214153907.770251000@chello.nl>
[-- Attachment #1: netvm-reserve.patch --]
[-- Type: text/plain, Size: 7304 bytes --]
Provide the basic infrastructure to reserve and charge/account network memory.
We provide the following reserve tree:
1) total network reserve
2) network TX reserve
3) protocol TX pages
4) network RX reserve
5) SKB data reserve
[1] is used to make all the network reserves a single subtree, for easy
manipulation.
[2] and [4] are merely for eastetic reasons.
The TX pages reserve [3] is assumed bounded by it being the upper bound of
memory that can be used for sending pages (not quite true, but good enough)
The SKB reserve [5] is an aggregate reserve, which is used to charge SKB data
against in the fallback path.
The consumers for these reserves are sockets marked with:
SOCK_MEMALLOC
Such sockets are to be used to service the VM (iow. to swap over). They
must be handled kernel side, exposing such a socket to user-space is a BUG.
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
---
include/net/sock.h | 35 +++++++++++++++-
net/Kconfig | 3 +
net/core/sock.c | 113 +++++++++++++++++++++++++++++++++++++++++++++++++++++
3 files changed, 150 insertions(+), 1 deletion(-)
Index: linux-2.6/include/net/sock.h
===================================================================
--- linux-2.6.orig/include/net/sock.h
+++ linux-2.6/include/net/sock.h
@@ -51,6 +51,7 @@
#include <linux/skbuff.h> /* struct sk_buff */
#include <linux/mm.h>
#include <linux/security.h>
+#include <linux/reserve.h>
#include <linux/filter.h>
@@ -403,6 +404,7 @@ enum sock_flags {
SOCK_RCVTSTAMPNS, /* %SO_TIMESTAMPNS setting */
SOCK_LOCALROUTE, /* route locally only, %SO_DONTROUTE setting */
SOCK_QUEUE_SHRUNK, /* write queue has been shrunk recently */
+ SOCK_MEMALLOC, /* the VM depends on us - make sure we're serviced */
};
static inline void sock_copy_flags(struct sock *nsk, struct sock *osk)
@@ -425,9 +427,40 @@ static inline int sock_flag(struct sock
return test_bit(flag, &sk->sk_flags);
}
+static inline int sk_has_memalloc(struct sock *sk)
+{
+ return sock_flag(sk, SOCK_MEMALLOC);
+}
+
+/*
+ * Guestimate the per request queue TX upper bound.
+ *
+ * Max packet size is 64k, and we need to reserve that much since the data
+ * might need to bounce it. Double it to be on the safe side.
+ */
+#define TX_RESERVE_PAGES DIV_ROUND_UP(2*65536, PAGE_SIZE)
+
+extern atomic_t memalloc_socks;
+
+extern struct mem_reserve net_rx_reserve;
+extern struct mem_reserve net_skb_reserve;
+
+static inline int sk_memalloc_socks(void)
+{
+ return atomic_read(&memalloc_socks);
+}
+
+extern int rx_emergency_get(int bytes);
+extern int rx_emergency_get_overcommit(int bytes);
+extern void rx_emergency_put(int bytes);
+
+extern int sk_adjust_memalloc(int socks, long tx_reserve_pages);
+extern int sk_set_memalloc(struct sock *sk);
+extern int sk_clear_memalloc(struct sock *sk);
+
static inline gfp_t sk_allocation(struct sock *sk, gfp_t gfp_mask)
{
- return gfp_mask;
+ return gfp_mask | (sk->sk_allocation & __GFP_MEMALLOC);
}
static inline void sk_acceptq_removed(struct sock *sk)
Index: linux-2.6/net/core/sock.c
===================================================================
--- linux-2.6.orig/net/core/sock.c
+++ linux-2.6/net/core/sock.c
@@ -112,6 +112,7 @@
#include <linux/tcp.h>
#include <linux/init.h>
#include <linux/highmem.h>
+#include <linux/reserve.h>
#include <asm/uaccess.h>
#include <asm/system.h>
@@ -213,6 +214,111 @@ __u32 sysctl_rmem_default __read_mostly
/* Maximal space eaten by iovec or ancilliary data plus some space */
int sysctl_optmem_max __read_mostly = sizeof(unsigned long)*(2*UIO_MAXIOV+512);
+atomic_t memalloc_socks;
+
+static struct mem_reserve net_reserve;
+struct mem_reserve net_rx_reserve;
+struct mem_reserve net_skb_reserve;
+static struct mem_reserve net_tx_reserve;
+static struct mem_reserve net_tx_pages;
+
+EXPORT_SYMBOL_GPL(net_rx_reserve); /* modular ipv6 only */
+EXPORT_SYMBOL_GPL(net_skb_reserve); /* modular ipv6 only */
+
+/*
+ * is there room for another emergency packet?
+ */
+static int __rx_emergency_get(int bytes, bool overcommit)
+{
+ return mem_reserve_kmalloc_charge(&net_skb_reserve, bytes, overcommit);
+}
+
+int rx_emergency_get(int bytes)
+{
+ return __rx_emergency_get(bytes, false);
+}
+
+int rx_emergency_get_overcommit(int bytes)
+{
+ return __rx_emergency_get(bytes, true);
+}
+
+void rx_emergency_put(int bytes)
+{
+ mem_reserve_kmalloc_charge(&net_skb_reserve, -bytes, 0);
+}
+
+/**
+ * sk_adjust_memalloc - adjust the global memalloc reserve for critical RX
+ * @socks: number of new %SOCK_MEMALLOC sockets
+ * @tx_resserve_pages: number of pages to (un)reserve for TX
+ *
+ * This function adjusts the memalloc reserve based on system demand.
+ * The RX reserve is a limit, and only added once, not for each socket.
+ *
+ * NOTE:
+ * @tx_reserve_pages is an upper-bound of memory used for TX hence
+ * we need not account the pages like we do for RX pages.
+ */
+int sk_adjust_memalloc(int socks, long tx_reserve_pages)
+{
+ int nr_socks;
+ int err;
+
+ err = mem_reserve_pages_add(&net_tx_pages, tx_reserve_pages);
+ if (err)
+ return err;
+
+ nr_socks = atomic_read(&memalloc_socks);
+ if (!nr_socks && socks > 0)
+ err = mem_reserve_connect(&net_reserve, &mem_reserve_root);
+ nr_socks = atomic_add_return(socks, &memalloc_socks);
+ if (!nr_socks && socks)
+ err = mem_reserve_disconnect(&net_reserve);
+
+ if (err)
+ mem_reserve_pages_add(&net_tx_pages, -tx_reserve_pages);
+
+ return err;
+}
+
+/**
+ * sk_set_memalloc - sets %SOCK_MEMALLOC
+ * @sk: socket to set it on
+ *
+ * Set %SOCK_MEMALLOC on a socket and increase the memalloc reserve
+ * accordingly.
+ */
+int sk_set_memalloc(struct sock *sk)
+{
+ int set = sock_flag(sk, SOCK_MEMALLOC);
+#ifndef CONFIG_NETVM
+ BUG();
+#endif
+ if (!set) {
+ int err = sk_adjust_memalloc(1, 0);
+ if (err)
+ return err;
+
+ sock_set_flag(sk, SOCK_MEMALLOC);
+ sk->sk_allocation |= __GFP_MEMALLOC;
+ }
+ return !set;
+}
+EXPORT_SYMBOL_GPL(sk_set_memalloc);
+
+int sk_clear_memalloc(struct sock *sk)
+{
+ int set = sock_flag(sk, SOCK_MEMALLOC);
+ if (set) {
+ sk_adjust_memalloc(-1, 0);
+ sock_reset_flag(sk, SOCK_MEMALLOC);
+ sk->sk_allocation &= ~__GFP_MEMALLOC;
+ }
+ return set;
+}
+EXPORT_SYMBOL_GPL(sk_clear_memalloc);
+
static int sock_set_timeout(long *timeo_p, char __user *optval, int optlen)
{
struct timeval tv;
@@ -952,6 +1058,7 @@ void sk_free(struct sock *sk)
{
struct sk_filter *filter;
+ sk_clear_memalloc(sk);
if (sk->sk_destruct)
sk->sk_destruct(sk);
@@ -1079,6 +1186,12 @@ void __init sk_init(void)
sysctl_wmem_max = 131071;
sysctl_rmem_max = 131071;
}
+
+ mem_reserve_init(&net_reserve, "total network reserve", NULL);
+ mem_reserve_init(&net_rx_reserve, "network RX reserve", &net_reserve);
+ mem_reserve_init(&net_skb_reserve, "SKB data reserve", &net_rx_reserve);
+ mem_reserve_init(&net_tx_reserve, "network TX reserve", &net_reserve);
+ mem_reserve_init(&net_tx_pages, "protocol TX pages", &net_tx_reserve);
}
/*
Index: linux-2.6/net/Kconfig
===================================================================
--- linux-2.6.orig/net/Kconfig
+++ linux-2.6/net/Kconfig
@@ -238,6 +238,9 @@ endmenu
source "net/rfkill/Kconfig"
source "net/9p/Kconfig"
+config NETVM
+ def_bool n
+
endif # if NET
endmenu # Networking
--
^ permalink raw reply
* [PATCH 08/29] mm: system wide ALLOC_NO_WATERMARK
From: Peter Zijlstra @ 2007-12-14 15:39 UTC (permalink / raw)
To: Linus Torvalds, Andrew Morton, linux-kernel, linux-mm, netdev,
trond.myklebust
Cc: Peter Zijlstra
In-Reply-To: <20071214153907.770251000@chello.nl>
[-- Attachment #1: global-ALLOC_NO_WATERMARKS.patch --]
[-- Type: text/plain, Size: 853 bytes --]
Change ALLOC_NO_WATERMARK page allocation such that the reserves are system
wide - which they are per setup_per_zone_pages_min(), when we scrape the
barrel, do it properly.
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
---
mm/page_alloc.c | 6 ++++++
1 file changed, 6 insertions(+)
Index: linux-2.6/mm/page_alloc.c
===================================================================
--- linux-2.6.orig/mm/page_alloc.c
+++ linux-2.6/mm/page_alloc.c
@@ -1638,6 +1638,12 @@ restart:
rebalance:
if (alloc_flags & ALLOC_NO_WATERMARKS) {
nofail_alloc:
+ /*
+ * break out of mempolicy boundaries
+ */
+ zonelist = NODE_DATA(numa_node_id())->node_zonelists +
+ gfp_zone(gfp_mask);
+
/* go through the zonelist yet again, ignoring mins */
page = get_page_from_freelist(gfp_mask, order, zonelist,
ALLOC_NO_WATERMARKS);
--
^ permalink raw reply
* [PATCH 03/29] mm: slb: add knowledge of reserve pages
From: Peter Zijlstra @ 2007-12-14 15:39 UTC (permalink / raw)
To: Linus Torvalds, Andrew Morton, linux-kernel, linux-mm, netdev,
trond.myklebust
Cc: Peter Zijlstra
In-Reply-To: <20071214153907.770251000@chello.nl>
[-- Attachment #1: reserve-slub.patch --]
[-- Type: text/plain, Size: 9728 bytes --]
Restrict objects from reserve slabs (ALLOC_NO_WATERMARKS) to allocation
contexts that are entitled to it. This is done to ensure reserve pages don't
leak out and get consumed.
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
---
include/linux/slub_def.h | 1
mm/slab.c | 59 +++++++++++++++++++++++++++++++++++++++--------
mm/slub.c | 27 ++++++++++++++++-----
3 files changed, 72 insertions(+), 15 deletions(-)
Index: linux-2.6/mm/slub.c
===================================================================
--- linux-2.6.orig/mm/slub.c
+++ linux-2.6/mm/slub.c
@@ -21,11 +21,12 @@
#include <linux/ctype.h>
#include <linux/kallsyms.h>
#include <linux/memory.h>
+#include "internal.h"
/*
* Lock order:
* 1. slab_lock(page)
- * 2. slab->list_lock
+ * 2. node->list_lock
*
* The slab_lock protects operations on the object of a particular
* slab and its metadata in the page struct. If the slab lock
@@ -1071,7 +1072,7 @@ static void setup_object(struct kmem_cac
}
static noinline struct page *new_slab(struct kmem_cache *s,
- gfp_t flags, int node)
+ gfp_t flags, int node, int *reserve)
{
struct page *page;
struct kmem_cache_node *n;
@@ -1087,6 +1088,7 @@ static noinline struct page *new_slab(st
if (!page)
goto out;
+ *reserve = page->reserve;
n = get_node(s, page_to_nid(page));
if (n)
atomic_long_inc(&n->nr_slabs);
@@ -1517,11 +1519,12 @@ static noinline unsigned long get_new_sl
{
struct kmem_cache_cpu *c = *pc;
struct page *page;
+ int reserve;
if (gfpflags & __GFP_WAIT)
local_irq_enable();
- page = new_slab(s, gfpflags, node);
+ page = new_slab(s, gfpflags, node, &reserve);
if (gfpflags & __GFP_WAIT)
local_irq_disable();
@@ -1530,6 +1533,7 @@ static noinline unsigned long get_new_sl
return 0;
*pc = c = get_cpu_slab(s, smp_processor_id());
+ c->reserve = reserve;
if (c->page)
flush_slab(s, c);
c->page = page;
@@ -1564,6 +1568,16 @@ static void *__slab_alloc(struct kmem_ca
local_irq_save(flags);
preempt_enable_no_resched();
#endif
+ if (unlikely(c->reserve)) {
+ /*
+ * If the current slab is a reserve slab and the current
+ * allocation context does not allow access to the reserves we
+ * must force an allocation to test the current levels.
+ */
+ if (!(gfp_to_alloc_flags(gfpflags) & ALLOC_NO_WATERMARKS))
+ goto grow_slab;
+ }
+
if (likely(c->page)) {
state = slab_lock(c->page);
@@ -1586,7 +1600,7 @@ load_freelist:
*/
VM_BUG_ON(c->page->freelist == c->page->end);
- if (unlikely(state & SLABDEBUG))
+ if (unlikely((state & SLABDEBUG) || c->reserve))
goto debug;
object = c->page->freelist;
@@ -1615,7 +1629,7 @@ grow_slab:
/* Perform debugging */
debug:
object = c->page->freelist;
- if (!alloc_debug_processing(s, c->page, object, addr))
+ if ((state & SLABDEBUG) && !alloc_debug_processing(s, c->page, object, addr))
goto another_slab;
c->page->inuse++;
@@ -2156,10 +2170,11 @@ static struct kmem_cache_node *early_kme
struct page *page;
struct kmem_cache_node *n;
unsigned long flags;
+ int reserve;
BUG_ON(kmalloc_caches->size < sizeof(struct kmem_cache_node));
- page = new_slab(kmalloc_caches, gfpflags, node);
+ page = new_slab(kmalloc_caches, gfpflags, node, &reserve);
BUG_ON(!page);
if (page_to_nid(page) != node) {
Index: linux-2.6/include/linux/slub_def.h
===================================================================
--- linux-2.6.orig/include/linux/slub_def.h
+++ linux-2.6/include/linux/slub_def.h
@@ -18,6 +18,7 @@ struct kmem_cache_cpu {
unsigned int offset; /* Freepointer offset (in word units) */
unsigned int objsize; /* Size of an object (from kmem_cache) */
unsigned int objects; /* Objects per slab (from kmem_cache) */
+ int reserve; /* Did the current page come from the reserve */
};
struct kmem_cache_node {
Index: linux-2.6/mm/slab.c
===================================================================
--- linux-2.6.orig/mm/slab.c
+++ linux-2.6/mm/slab.c
@@ -115,6 +115,8 @@
#include <asm/tlbflush.h>
#include <asm/page.h>
+#include "internal.h"
+
/*
* DEBUG - 1 for kmem_cache_create() to honour; SLAB_RED_ZONE & SLAB_POISON.
* 0 for faster, smaller code (especially in the critical paths).
@@ -265,7 +267,8 @@ struct array_cache {
unsigned int avail;
unsigned int limit;
unsigned int batchcount;
- unsigned int touched;
+ unsigned int touched:1,
+ reserve:1;
spinlock_t lock;
void *entry[]; /*
* Must have this definition in here for the proper
@@ -762,6 +765,27 @@ static inline struct array_cache *cpu_ca
return cachep->array[smp_processor_id()];
}
+/*
+ * If the last page came from the reserves, and the current allocation context
+ * does not have access to them, force an allocation to test the watermarks.
+ */
+static inline int slab_force_alloc(struct kmem_cache *cachep, gfp_t flags)
+{
+ if (unlikely(cpu_cache_get(cachep)->reserve) &&
+ !(gfp_to_alloc_flags(flags) & ALLOC_NO_WATERMARKS))
+ return 1;
+
+ return 0;
+}
+
+static inline void slab_set_reserve(struct kmem_cache *cachep, int reserve)
+{
+ struct array_cache *ac = cpu_cache_get(cachep);
+
+ if (unlikely(ac->reserve != reserve))
+ ac->reserve = reserve;
+}
+
static inline struct kmem_cache *__find_general_cachep(size_t size,
gfp_t gfpflags)
{
@@ -961,6 +985,7 @@ static struct array_cache *alloc_arrayca
nc->limit = entries;
nc->batchcount = batchcount;
nc->touched = 0;
+ nc->reserve = 0;
spin_lock_init(&nc->lock);
}
return nc;
@@ -1650,7 +1675,8 @@ __initcall(cpucache_init);
* did not request dmaable memory, we might get it, but that
* would be relatively rare and ignorable.
*/
-static void *kmem_getpages(struct kmem_cache *cachep, gfp_t flags, int nodeid)
+static void *kmem_getpages(struct kmem_cache *cachep, gfp_t flags, int nodeid,
+ int *reserve)
{
struct page *page;
int nr_pages;
@@ -1672,6 +1698,7 @@ static void *kmem_getpages(struct kmem_c
if (!page)
return NULL;
+ *reserve = page->reserve;
nr_pages = (1 << cachep->gfporder);
if (cachep->flags & SLAB_RECLAIM_ACCOUNT)
add_zone_page_state(page_zone(page),
@@ -2116,6 +2143,7 @@ static int __init_refok setup_cpu_cache(
cpu_cache_get(cachep)->limit = BOOT_CPUCACHE_ENTRIES;
cpu_cache_get(cachep)->batchcount = 1;
cpu_cache_get(cachep)->touched = 0;
+ cpu_cache_get(cachep)->reserve = 0;
cachep->batchcount = 1;
cachep->limit = BOOT_CPUCACHE_ENTRIES;
return 0;
@@ -2764,6 +2792,7 @@ static int cache_grow(struct kmem_cache
size_t offset;
gfp_t local_flags;
struct kmem_list3 *l3;
+ int reserve;
/*
* Be lazy and only check for valid flags here, keeping it out of the
@@ -2802,7 +2831,7 @@ static int cache_grow(struct kmem_cache
* 'nodeid'.
*/
if (!objp)
- objp = kmem_getpages(cachep, local_flags, nodeid);
+ objp = kmem_getpages(cachep, local_flags, nodeid, &reserve);
if (!objp)
goto failed;
@@ -2820,6 +2849,7 @@ static int cache_grow(struct kmem_cache
if (local_flags & __GFP_WAIT)
local_irq_disable();
check_irq_off();
+ slab_set_reserve(cachep, reserve);
spin_lock(&l3->list_lock);
/* Make slab active. */
@@ -2954,7 +2984,7 @@ bad:
#define check_slabp(x,y) do { } while(0)
#endif
-static void *cache_alloc_refill(struct kmem_cache *cachep, gfp_t flags)
+static void *cache_alloc_refill(struct kmem_cache *cachep, gfp_t flags, int must_refill)
{
int batchcount;
struct kmem_list3 *l3;
@@ -2964,6 +2994,8 @@ static void *cache_alloc_refill(struct k
node = numa_node_id();
check_irq_off();
+ if (unlikely(must_refill))
+ goto force_grow;
ac = cpu_cache_get(cachep);
retry:
batchcount = ac->batchcount;
@@ -3032,11 +3064,14 @@ alloc_done:
if (unlikely(!ac->avail)) {
int x;
+force_grow:
x = cache_grow(cachep, flags | GFP_THISNODE, node, NULL);
/* cache_grow can reenable interrupts, then ac could change. */
ac = cpu_cache_get(cachep);
- if (!x && ac->avail == 0) /* no objects in sight? abort */
+
+ /* no objects in sight? abort */
+ if (!x && (ac->avail == 0 || must_refill))
return NULL;
if (!ac->avail) /* objects refilled by interrupt? */
@@ -3191,17 +3226,18 @@ static inline void *____cache_alloc(stru
{
void *objp;
struct array_cache *ac;
+ int must_refill = slab_force_alloc(cachep, flags);
check_irq_off();
ac = cpu_cache_get(cachep);
- if (likely(ac->avail)) {
+ if (likely(ac->avail && !must_refill)) {
STATS_INC_ALLOCHIT(cachep);
ac->touched = 1;
objp = ac->entry[--ac->avail];
} else {
STATS_INC_ALLOCMISS(cachep);
- objp = cache_alloc_refill(cachep, flags);
+ objp = cache_alloc_refill(cachep, flags, must_refill);
}
return objp;
}
@@ -3243,7 +3279,7 @@ static void *fallback_alloc(struct kmem_
gfp_t local_flags;
struct zone **z;
void *obj = NULL;
- int nid;
+ int nid, reserve;
if (flags & __GFP_THISNODE)
return NULL;
@@ -3277,10 +3313,11 @@ retry:
if (local_flags & __GFP_WAIT)
local_irq_enable();
kmem_flagcheck(cache, flags);
- obj = kmem_getpages(cache, flags, -1);
+ obj = kmem_getpages(cache, flags, -1, &reserve);
if (local_flags & __GFP_WAIT)
local_irq_disable();
if (obj) {
+ slab_set_reserve(cache, reserve);
/*
* Insert into the appropriate per node queues
*/
@@ -3319,6 +3356,9 @@ static void *____cache_alloc_node(struct
l3 = cachep->nodelists[nodeid];
BUG_ON(!l3);
+ if (unlikely(slab_force_alloc(cachep, flags)))
+ goto force_grow;
+
retry:
check_irq_off();
spin_lock(&l3->list_lock);
@@ -3356,6 +3396,7 @@ retry:
must_grow:
spin_unlock(&l3->list_lock);
+force_grow:
x = cache_grow(cachep, flags | GFP_THISNODE, nodeid, NULL);
if (x)
goto retry;
--
^ permalink raw reply
* [PATCH 27/29] nfs: disable data cache revalidation for swapfiles
From: Peter Zijlstra @ 2007-12-14 15:39 UTC (permalink / raw)
To: Linus Torvalds, Andrew Morton, linux-kernel, linux-mm, netdev,
trond.myklebust
Cc: Peter Zijlstra
In-Reply-To: <20071214153907.770251000@chello.nl>
[-- Attachment #1: nfs-swapper.patch --]
[-- Type: text/plain, Size: 5230 bytes --]
Do as Trond suggested:
http://lkml.org/lkml/2006/8/25/348
Disable NFS data cache revalidation on swap files since it doesn't really
make sense to have other clients change the file while you are using it.
Thereby we can stop setting PG_private on swap pages, since there ought to
be no further races with invalidate_inode_pages2() to deal with.
And since we cannot set PG_private we cannot use page->private (which is
already used by PG_swapcache pages anyway) to store the nfs_page. Thus
augment the new nfs_page_find_request logic.
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
---
fs/nfs/inode.c | 6 ++++
fs/nfs/write.c | 73 ++++++++++++++++++++++++++++++++++++++++++++++-----------
2 files changed, 65 insertions(+), 14 deletions(-)
Index: linux-2.6/fs/nfs/inode.c
===================================================================
--- linux-2.6.orig/fs/nfs/inode.c
+++ linux-2.6/fs/nfs/inode.c
@@ -758,6 +758,12 @@ int nfs_revalidate_mapping_nolock(struct
struct nfs_inode *nfsi = NFS_I(inode);
int ret = 0;
+ /*
+ * swapfiles are not supposed to be shared.
+ */
+ if (IS_SWAPFILE(inode))
+ goto out;
+
if ((nfsi->cache_validity & NFS_INO_REVAL_PAGECACHE)
|| nfs_attribute_timeout(inode) || NFS_STALE(inode)) {
ret = __nfs_revalidate_inode(NFS_SERVER(inode), inode);
Index: linux-2.6/fs/nfs/write.c
===================================================================
--- linux-2.6.orig/fs/nfs/write.c
+++ linux-2.6/fs/nfs/write.c
@@ -112,25 +112,62 @@ static void nfs_context_set_write_error(
set_bit(NFS_CONTEXT_ERROR_WRITE, &ctx->flags);
}
-static struct nfs_page *nfs_page_find_request_locked(struct page *page)
+static struct nfs_page *
+__nfs_page_find_request_locked(struct nfs_inode *nfsi, struct page *page, int get)
{
struct nfs_page *req = NULL;
- if (PagePrivate(page)) {
+ if (PagePrivate(page))
req = (struct nfs_page *)page_private(page);
- if (req != NULL)
- kref_get(&req->wb_kref);
- }
+ else if (unlikely(PageSwapCache(page)))
+ req = radix_tree_lookup(&nfsi->nfs_page_tree, page_file_index(page));
+
+ if (get && req)
+ kref_get(&req->wb_kref);
+
return req;
}
+static inline struct nfs_page *
+nfs_page_find_request_locked(struct nfs_inode *nfsi, struct page *page)
+{
+ return __nfs_page_find_request_locked(nfsi, page, 1);
+}
+
+static int __nfs_page_has_request(struct page *page)
+{
+ struct inode *inode = page_file_mapping(page)->host;
+ struct nfs_page *req = NULL;
+
+ spin_lock(&inode->i_lock);
+ req = __nfs_page_find_request_locked(NFS_I(inode), page, 0);
+ spin_unlock(&inode->i_lock);
+
+ /*
+ * hole here plugged by the caller holding onto PG_locked
+ */
+
+ return req != NULL;
+}
+
+static inline int nfs_page_has_request(struct page *page)
+{
+ if (PagePrivate(page))
+ return 1;
+
+ if (unlikely(PageSwapCache(page)))
+ return __nfs_page_has_request(page);
+
+ return 0;
+}
+
static struct nfs_page *nfs_page_find_request(struct page *page)
{
struct inode *inode = page_file_mapping(page)->host;
struct nfs_page *req = NULL;
spin_lock(&inode->i_lock);
- req = nfs_page_find_request_locked(page);
+ req = nfs_page_find_request_locked(NFS_I(inode), page);
spin_unlock(&inode->i_lock);
return req;
}
@@ -253,7 +290,7 @@ static int nfs_page_async_flush(struct n
spin_lock(&inode->i_lock);
for(;;) {
- req = nfs_page_find_request_locked(page);
+ req = nfs_page_find_request_locked(nfsi, page);
if (req == NULL) {
spin_unlock(&inode->i_lock);
return 0;
@@ -370,8 +407,14 @@ static void nfs_inode_add_request(struct
if (nfs_have_delegation(inode, FMODE_WRITE))
nfsi->change_attr++;
}
- SetPagePrivate(req->wb_page);
- set_page_private(req->wb_page, (unsigned long)req);
+ /*
+ * Swap-space should not get truncated. Hence no need to plug the race
+ * with invalidate/truncate.
+ */
+ if (likely(!PageSwapCache(req->wb_page))) {
+ SetPagePrivate(req->wb_page);
+ set_page_private(req->wb_page, (unsigned long)req);
+ }
nfsi->npages++;
kref_get(&req->wb_kref);
}
@@ -387,8 +430,10 @@ static void nfs_inode_remove_request(str
BUG_ON (!NFS_WBACK_BUSY(req));
spin_lock(&inode->i_lock);
- set_page_private(req->wb_page, 0);
- ClearPagePrivate(req->wb_page);
+ if (likely(!PageSwapCache(req->wb_page))) {
+ set_page_private(req->wb_page, 0);
+ ClearPagePrivate(req->wb_page);
+ }
radix_tree_delete(&nfsi->nfs_page_tree, req->wb_index);
nfsi->npages--;
if (!nfsi->npages) {
@@ -594,7 +639,7 @@ static struct nfs_page * nfs_update_requ
}
spin_lock(&inode->i_lock);
- req = nfs_page_find_request_locked(page);
+ req = nfs_page_find_request_locked(NFS_I(inode), page);
if (req) {
if (!nfs_lock_request_dontget(req)) {
int error;
@@ -1426,7 +1471,7 @@ int nfs_wb_page_cancel(struct inode *ino
if (ret < 0)
goto out;
}
- if (!PagePrivate(page))
+ if (!nfs_page_has_request(page))
return 0;
ret = nfs_sync_mapping_wait(page_file_mapping(page), &wbc, FLUSH_INVALIDATE);
out:
@@ -1453,7 +1498,7 @@ static int nfs_wb_page_priority(struct i
if (ret < 0)
goto out;
}
- if (!PagePrivate(page))
+ if (!nfs_page_has_request(page))
return 0;
ret = nfs_sync_mapping_wait(page_file_mapping(page), &wbc, how);
if (ret >= 0)
--
^ permalink raw reply
* [PATCH 24/29] mm: methods for teaching filesystems about PG_swapcache pages
From: Peter Zijlstra @ 2007-12-14 15:39 UTC (permalink / raw)
To: Linus Torvalds, Andrew Morton, linux-kernel, linux-mm, netdev,
trond.myklebust
Cc: Peter Zijlstra
In-Reply-To: <20071214153907.770251000@chello.nl>
[-- Attachment #1: mm-page_file_methods.patch --]
[-- Type: text/plain, Size: 2789 bytes --]
In order to teach filesystems to handle swap cache pages, two new page
functions are introduced:
pgoff_t page_file_index(struct page *);
struct address_space *page_file_mapping(struct page *);
page_file_index - gives the offset of this page in the file in PAGE_CACHE_SIZE
blocks. Like page->index is for mapped pages, this function also gives the
correct index for PG_swapcache pages.
page_file_mapping - gives the mapping backing the actual page; that is for
swap cache pages it will give swap_file->f_mapping.
page_offset() is modified to use page_file_index(), so that it will give the
expected result, even for PG_swapcache pages.
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
---
include/linux/mm.h | 26 ++++++++++++++++++++++++++
include/linux/pagemap.h | 2 +-
2 files changed, 27 insertions(+), 1 deletion(-)
Index: linux-2.6/include/linux/mm.h
===================================================================
--- linux-2.6.orig/include/linux/mm.h
+++ linux-2.6/include/linux/mm.h
@@ -14,6 +14,7 @@
#include <linux/mm_types.h>
#include <linux/security.h>
#include <linux/swap.h>
+#include <linux/fs.h>
struct mempolicy;
struct anon_vma;
@@ -608,6 +609,16 @@ static inline struct swap_info_struct *p
return get_swap_info_struct(swp_type(swap));
}
+static inline
+struct address_space *page_file_mapping(struct page *page)
+{
+#ifdef CONFIG_SWAP_FILE
+ if (unlikely(PageSwapCache(page)))
+ return page_swap_info(page)->swap_file->f_mapping;
+#endif
+ return page->mapping;
+}
+
static inline int PageAnon(struct page *page)
{
return ((unsigned long)page->mapping & PAGE_MAPPING_ANON) != 0;
@@ -625,6 +636,21 @@ static inline pgoff_t page_index(struct
}
/*
+ * Return the file index of the page. Regular pagecache pages use ->index
+ * whereas swapcache pages use swp_offset(->private)
+ */
+static inline pgoff_t page_file_index(struct page *page)
+{
+#ifdef CONFIG_SWAP_FILE
+ if (unlikely(PageSwapCache(page))) {
+ swp_entry_t swap = { .val = page_private(page) };
+ return swp_offset(swap);
+ }
+#endif
+ return page->index;
+}
+
+/*
* The atomic page->_mapcount, like _count, starts from -1:
* so that transitions both from it and to it can be tracked,
* using atomic_inc_and_test and atomic_add_negative(-1).
Index: linux-2.6/include/linux/pagemap.h
===================================================================
--- linux-2.6.orig/include/linux/pagemap.h
+++ linux-2.6/include/linux/pagemap.h
@@ -145,7 +145,7 @@ extern void __remove_from_page_cache(str
*/
static inline loff_t page_offset(struct page *page)
{
- return ((loff_t)page->index) << PAGE_CACHE_SHIFT;
+ return ((loff_t)page_file_index(page)) << PAGE_CACHE_SHIFT;
}
static inline pgoff_t linear_page_index(struct vm_area_struct *vma,
--
^ permalink raw reply
* [PATCH 21/29] netvm: skb processing
From: Peter Zijlstra @ 2007-12-14 15:39 UTC (permalink / raw)
To: Linus Torvalds, Andrew Morton, linux-kernel, linux-mm, netdev,
trond.myklebust
Cc: Peter Zijlstra
In-Reply-To: <20071214153907.770251000@chello.nl>
[-- Attachment #1: netvm.patch --]
[-- Type: text/plain, Size: 4835 bytes --]
In order to make sure emergency packets receive all memory needed to proceed
ensure processing of emergency SKBs happens under PF_MEMALLOC.
Use the (new) sk_backlog_rcv() wrapper to ensure this for backlog processing.
Skip taps, since those are user-space again.
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
---
include/net/sock.h | 5 ++++
net/core/dev.c | 59 +++++++++++++++++++++++++++++++++++++++++++++++------
net/core/sock.c | 18 ++++++++++++++++
3 files changed, 76 insertions(+), 6 deletions(-)
Index: linux-2.6/net/core/dev.c
===================================================================
--- linux-2.6.orig/net/core/dev.c
+++ linux-2.6/net/core/dev.c
@@ -2008,6 +2008,30 @@ out:
}
#endif
+/*
+ * Filter the protocols for which the reserves are adequate.
+ *
+ * Before adding a protocol make sure that it is either covered by the existing
+ * reserves, or add reserves covering the memory need of the new protocol's
+ * packet processing.
+ */
+static int skb_emergency_protocol(struct sk_buff *skb)
+{
+ if (skb_emergency(skb))
+ switch(skb->protocol) {
+ case __constant_htons(ETH_P_ARP):
+ case __constant_htons(ETH_P_IP):
+ case __constant_htons(ETH_P_IPV6):
+ case __constant_htons(ETH_P_8021Q):
+ break;
+
+ default:
+ return 0;
+ }
+
+ return 1;
+}
+
/**
* netif_receive_skb - process receive buffer from network
* @skb: buffer to process
@@ -2029,10 +2053,23 @@ int netif_receive_skb(struct sk_buff *sk
struct net_device *orig_dev;
int ret = NET_RX_DROP;
__be16 type;
+ unsigned long pflags = current->flags;
+
+ /* Emergency skb are special, they should
+ * - be delivered to SOCK_MEMALLOC sockets only
+ * - stay away from userspace
+ * - have bounded memory usage
+ *
+ * Use PF_MEMALLOC as a poor mans memory pool - the grouping kind.
+ * This saves us from propagating the allocation context down to all
+ * allocation sites.
+ */
+ if (skb_emergency(skb))
+ current->flags |= PF_MEMALLOC;
/* if we've gotten here through NAPI, check netpoll */
if (netpoll_receive_skb(skb))
- return NET_RX_DROP;
+ goto out;
if (!skb->tstamp.tv64)
net_timestamp(skb);
@@ -2043,7 +2080,7 @@ int netif_receive_skb(struct sk_buff *sk
orig_dev = skb_bond(skb);
if (!orig_dev)
- return NET_RX_DROP;
+ goto out;
__get_cpu_var(netdev_rx_stat).total++;
@@ -2062,6 +2099,9 @@ int netif_receive_skb(struct sk_buff *sk
}
#endif
+ if (skb_emergency(skb))
+ goto skip_taps;
+
list_for_each_entry_rcu(ptype, &ptype_all, list) {
if (!ptype->dev || ptype->dev == skb->dev) {
if (pt_prev)
@@ -2070,19 +2110,23 @@ int netif_receive_skb(struct sk_buff *sk
}
}
+skip_taps:
#ifdef CONFIG_NET_CLS_ACT
skb = handle_ing(skb, &pt_prev, &ret, orig_dev);
if (!skb)
- goto out;
+ goto unlock;
ncls:
#endif
+ if (!skb_emergency_protocol(skb))
+ goto drop;
+
skb = handle_bridge(skb, &pt_prev, &ret, orig_dev);
if (!skb)
- goto out;
+ goto unlock;
skb = handle_macvlan(skb, &pt_prev, &ret, orig_dev);
if (!skb)
- goto out;
+ goto unlock;
type = skb->protocol;
list_for_each_entry_rcu(ptype,
@@ -2098,6 +2142,7 @@ ncls:
if (pt_prev) {
ret = pt_prev->func(skb, skb->dev, pt_prev, orig_dev);
} else {
+drop:
kfree_skb(skb);
/* Jamal, now you will not able to escape explaining
* me how you were going to use this. :-)
@@ -2105,8 +2150,10 @@ ncls:
ret = NET_RX_DROP;
}
-out:
+unlock:
rcu_read_unlock();
+out:
+ tsk_restore_flags(current, pflags, PF_MEMALLOC);
return ret;
}
Index: linux-2.6/include/net/sock.h
===================================================================
--- linux-2.6.orig/include/net/sock.h
+++ linux-2.6/include/net/sock.h
@@ -529,8 +529,13 @@ static inline void sk_add_backlog(struct
skb->next = NULL;
}
+extern int __sk_backlog_rcv(struct sock *sk, struct sk_buff *skb);
+
static inline int sk_backlog_rcv(struct sock *sk, struct sk_buff *skb)
{
+ if (skb_emergency(skb))
+ return __sk_backlog_rcv(sk, skb);
+
return sk->sk_backlog_rcv(sk, skb);
}
Index: linux-2.6/net/core/sock.c
===================================================================
--- linux-2.6.orig/net/core/sock.c
+++ linux-2.6/net/core/sock.c
@@ -319,6 +319,24 @@ int sk_clear_memalloc(struct sock *sk)
}
EXPORT_SYMBOL_GPL(sk_clear_memalloc);
+#ifdef CONFIG_NETVM
+int __sk_backlog_rcv(struct sock *sk, struct sk_buff *skb)
+{
+ int ret;
+ unsigned long pflags = current->flags;
+
+ /* these should have been dropped before queueing */
+ BUG_ON(!sk_has_memalloc(sk));
+
+ current->flags |= PF_MEMALLOC;
+ ret = sk->sk_backlog_rcv(sk, skb);
+ tsk_restore_flags(current, pflags, PF_MEMALLOC);
+
+ return ret;
+}
+EXPORT_SYMBOL(__sk_backlog_rcv);
+#endif
+
static int sock_set_timeout(long *timeo_p, char __user *optval, int optlen)
{
struct timeval tv;
--
^ permalink raw reply
* [PATCH 19/29] netvm: prevent a TCP specific deadlock
From: Peter Zijlstra @ 2007-12-14 15:39 UTC (permalink / raw)
To: Linus Torvalds, Andrew Morton, linux-kernel, linux-mm, netdev,
trond.myklebust
Cc: Peter Zijlstra
In-Reply-To: <20071214153907.770251000@chello.nl>
[-- Attachment #1: netvm-tcp-deadlock.patch --]
[-- Type: text/plain, Size: 2442 bytes --]
It could happen that all !SOCK_MEMALLOC sockets have buffered so much data
that we're over the global rmem limit. This will prevent SOCK_MEMALLOC buffers
from receiving data, which will prevent userspace from running, which is needed
to reduce the buffered data.
Fix this by exempting the SOCK_MEMALLOC sockets from the rmem limit.
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
---
include/net/sock.h | 7 ++++---
net/core/stream.c | 5 +++--
2 files changed, 7 insertions(+), 5 deletions(-)
Index: linux-2.6/include/net/sock.h
===================================================================
--- linux-2.6.orig/include/net/sock.h
+++ linux-2.6/include/net/sock.h
@@ -756,7 +756,8 @@ static inline struct inode *SOCK_INODE(s
}
extern void __sk_stream_mem_reclaim(struct sock *sk);
-extern int sk_stream_mem_schedule(struct sock *sk, int size, int kind);
+extern int sk_stream_mem_schedule(struct sock *sk, struct sk_buff *skb,
+ int size, int kind);
#define SK_STREAM_MEM_QUANTUM ((int)PAGE_SIZE)
@@ -774,13 +775,13 @@ static inline void sk_stream_mem_reclaim
static inline int sk_stream_rmem_schedule(struct sock *sk, struct sk_buff *skb)
{
return (int)skb->truesize <= sk->sk_forward_alloc ||
- sk_stream_mem_schedule(sk, skb->truesize, 1);
+ sk_stream_mem_schedule(sk, skb, skb->truesize, 1);
}
static inline int sk_stream_wmem_schedule(struct sock *sk, int size)
{
return size <= sk->sk_forward_alloc ||
- sk_stream_mem_schedule(sk, size, 0);
+ sk_stream_mem_schedule(sk, NULL, size, 0);
}
/* Used by processes to "lock" a socket state, so that
Index: linux-2.6/net/core/stream.c
===================================================================
--- linux-2.6.orig/net/core/stream.c
+++ linux-2.6/net/core/stream.c
@@ -207,7 +207,7 @@ void __sk_stream_mem_reclaim(struct sock
EXPORT_SYMBOL(__sk_stream_mem_reclaim);
-int sk_stream_mem_schedule(struct sock *sk, int size, int kind)
+int sk_stream_mem_schedule(struct sock *sk, struct sk_buff *skb, int size, int kind)
{
int amt = sk_stream_pages(size);
struct proto *prot = sk->sk_prot;
@@ -225,7 +225,8 @@ int sk_stream_mem_schedule(struct sock *
/* Over hard limit. */
if (atomic_read(prot->memory_allocated) > prot->sysctl_mem[2]) {
prot->enter_memory_pressure();
- goto suppress_allocation;
+ if (!skb || (skb && !skb_emergency(skb)))
+ goto suppress_allocation;
}
/* Under pressure. */
--
^ permalink raw reply
* [PATCH 11/29] selinux: tag avc cache alloc as non-critical
From: Peter Zijlstra @ 2007-12-14 15:39 UTC (permalink / raw)
To: Linus Torvalds, Andrew Morton, linux-kernel, linux-mm, netdev,
trond.myklebust
Cc: Peter Zijlstra, James Morris
In-Reply-To: <20071214153907.770251000@chello.nl>
[-- Attachment #1: mm-selinux-emergency.patch --]
[-- Type: text/plain, Size: 774 bytes --]
Failing to allocate a cache entry will only harm performance not correctness.
Do not consume valuable reserve pages for something like that.
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
Acked-by: James Morris <jmorris@namei.org>
---
security/selinux/avc.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
Index: linux-2.6-2/security/selinux/avc.c
===================================================================
--- linux-2.6-2.orig/security/selinux/avc.c
+++ linux-2.6-2/security/selinux/avc.c
@@ -334,7 +334,7 @@ static struct avc_node *avc_alloc_node(v
{
struct avc_node *node;
- node = kmem_cache_zalloc(avc_node_cachep, GFP_ATOMIC);
+ node = kmem_cache_zalloc(avc_node_cachep, GFP_ATOMIC|__GFP_NOMEMALLOC);
if (!node)
goto out;
--
^ permalink raw reply
* [PATCH 17/29] netvm: hook skb allocation to reserves
From: Peter Zijlstra @ 2007-12-14 15:39 UTC (permalink / raw)
To: Linus Torvalds, Andrew Morton, linux-kernel, linux-mm, netdev,
trond.myklebust
Cc: Peter Zijlstra
In-Reply-To: <20071214153907.770251000@chello.nl>
[-- Attachment #1: netvm-skbuff-reserve.patch --]
[-- Type: text/plain, Size: 14946 bytes --]
Change the skb allocation api to indicate RX usage and use this to fall back to
the reserve when needed. SKBs allocated from the reserve are tagged in
skb->emergency.
Teach all other skb ops about emergency skbs and the reserve accounting.
Use the (new) packet split API to allocate and track fragment pages from the
emergency reserve. Do this using an atomic counter in page->index. This is
needed because the fragments have a different sharing semantic than that
indicated by skb_shinfo()->dataref.
Note that the decision to distinguish between regular and emergency SKBs allows
the accounting overhead to be limited to the later kind.
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
---
include/linux/mm_types.h | 1
include/linux/skbuff.h | 26 +++++--
net/core/skbuff.c | 173 +++++++++++++++++++++++++++++++++++++++++------
3 files changed, 174 insertions(+), 26 deletions(-)
Index: linux-2.6/include/linux/skbuff.h
===================================================================
--- linux-2.6.orig/include/linux/skbuff.h
+++ linux-2.6/include/linux/skbuff.h
@@ -314,7 +314,9 @@ struct sk_buff {
__u16 tc_verd; /* traffic control verdict */
#endif
#endif
- /* 2 byte hole */
+ __u8 emergency:1;
+ /* 7 bit hole */
+ /* 1 byte hole */
#ifdef CONFIG_NET_DMA
dma_cookie_t dma_cookie;
@@ -345,10 +347,22 @@ struct sk_buff {
#include <asm/system.h>
+#define SKB_ALLOC_FCLONE 0x01
+#define SKB_ALLOC_RX 0x02
+
+static inline bool skb_emergency(const struct sk_buff *skb)
+{
+#ifdef CONFIG_NETVM
+ return unlikely(skb->emergency);
+#else
+ return false;
+#endif
+}
+
extern void kfree_skb(struct sk_buff *skb);
extern void __kfree_skb(struct sk_buff *skb);
extern struct sk_buff *__alloc_skb(unsigned int size,
- gfp_t priority, int fclone, int node);
+ gfp_t priority, int flags, int node);
static inline struct sk_buff *alloc_skb(unsigned int size,
gfp_t priority)
{
@@ -358,7 +372,7 @@ static inline struct sk_buff *alloc_skb(
static inline struct sk_buff *alloc_skb_fclone(unsigned int size,
gfp_t priority)
{
- return __alloc_skb(size, priority, 1, -1);
+ return __alloc_skb(size, priority, SKB_ALLOC_FCLONE, -1);
}
extern struct sk_buff *skb_morph(struct sk_buff *dst, struct sk_buff *src);
@@ -1303,7 +1317,8 @@ static inline void __skb_queue_purge(str
static inline struct sk_buff *__dev_alloc_skb(unsigned int length,
gfp_t gfp_mask)
{
- struct sk_buff *skb = alloc_skb(length + NET_SKB_PAD, gfp_mask);
+ struct sk_buff *skb =
+ __alloc_skb(length + NET_SKB_PAD, gfp_mask, SKB_ALLOC_RX, -1);
if (likely(skb))
skb_reserve(skb, NET_SKB_PAD);
return skb;
@@ -1349,6 +1364,7 @@ static inline struct sk_buff *netdev_all
}
extern struct page *__netdev_alloc_page(struct net_device *dev, gfp_t gfp_mask);
+extern void __netdev_free_page(struct net_device *dev, struct page *page);
/**
* netdev_alloc_page - allocate a page for ps-rx on a specific device
@@ -1365,7 +1381,7 @@ static inline struct page *netdev_alloc_
static inline void netdev_free_page(struct net_device *dev, struct page *page)
{
- __free_page(page);
+ __netdev_free_page(dev, page);
}
/**
Index: linux-2.6/net/core/skbuff.c
===================================================================
--- linux-2.6.orig/net/core/skbuff.c
+++ linux-2.6/net/core/skbuff.c
@@ -179,21 +179,28 @@ EXPORT_SYMBOL(skb_truesize_bug);
* %GFP_ATOMIC.
*/
struct sk_buff *__alloc_skb(unsigned int size, gfp_t gfp_mask,
- int fclone, int node)
+ int flags, int node)
{
struct kmem_cache *cache;
struct skb_shared_info *shinfo;
struct sk_buff *skb;
u8 *data;
+ int emergency = 0, memalloc = sk_memalloc_socks();
- cache = fclone ? skbuff_fclone_cache : skbuff_head_cache;
+ size = SKB_DATA_ALIGN(size);
+ cache = (flags & SKB_ALLOC_FCLONE)
+ ? skbuff_fclone_cache : skbuff_head_cache;
+#ifdef CONFIG_NETVM
+ if (memalloc && (flags & SKB_ALLOC_RX))
+ gfp_mask |= __GFP_NOMEMALLOC|__GFP_NOWARN;
+retry_alloc:
+#endif
/* Get the HEAD */
skb = kmem_cache_alloc_node(cache, gfp_mask & ~__GFP_DMA, node);
if (!skb)
- goto out;
+ goto noskb;
- size = SKB_DATA_ALIGN(size);
data = kmalloc_node_track_caller(size + sizeof(struct skb_shared_info),
gfp_mask, node);
if (!data)
@@ -203,6 +210,7 @@ struct sk_buff *__alloc_skb(unsigned int
* See comment in sk_buff definition, just before the 'tail' member
*/
memset(skb, 0, offsetof(struct sk_buff, tail));
+ skb->emergency = emergency;
skb->truesize = size + sizeof(struct sk_buff);
atomic_set(&skb->users, 1);
skb->head = data;
@@ -219,7 +227,7 @@ struct sk_buff *__alloc_skb(unsigned int
shinfo->ip6_frag_id = 0;
shinfo->frag_list = NULL;
- if (fclone) {
+ if (flags & SKB_ALLOC_FCLONE) {
struct sk_buff *child = skb + 1;
atomic_t *fclone_ref = (atomic_t *) (child + 1);
@@ -227,12 +235,31 @@ struct sk_buff *__alloc_skb(unsigned int
atomic_set(fclone_ref, 1);
child->fclone = SKB_FCLONE_UNAVAILABLE;
+ child->emergency = skb->emergency;
}
out:
return skb;
+
nodata:
kmem_cache_free(cache, skb);
skb = NULL;
+noskb:
+#ifdef CONFIG_NETVM
+ /* Attempt emergency allocation when RX skb. */
+ if (likely(!(flags & SKB_ALLOC_RX) || !memalloc))
+ goto out;
+
+ if (!emergency) {
+ if (rx_emergency_get(size)) {
+ gfp_mask &= ~(__GFP_NOMEMALLOC|__GFP_NOWARN);
+ gfp_mask |= __GFP_MEMALLOC;
+ emergency = 1;
+ goto retry_alloc;
+ }
+ } else
+ rx_emergency_put(size);
+#endif
+
goto out;
}
@@ -255,7 +282,7 @@ struct sk_buff *__netdev_alloc_skb(struc
int node = dev->dev.parent ? dev_to_node(dev->dev.parent) : -1;
struct sk_buff *skb;
- skb = __alloc_skb(length + NET_SKB_PAD, gfp_mask, 0, node);
+ skb = __alloc_skb(length + NET_SKB_PAD, gfp_mask, SKB_ALLOC_RX, node);
if (likely(skb)) {
skb_reserve(skb, NET_SKB_PAD);
skb->dev = dev;
@@ -268,10 +295,34 @@ struct page *__netdev_alloc_page(struct
int node = dev->dev.parent ? dev_to_node(dev->dev.parent) : -1;
struct page *page;
+#ifdef CONFIG_NETVM
+ gfp_mask |= __GFP_NOMEMALLOC | __GFP_NOWARN;
+#endif
+
page = alloc_pages_node(node, gfp_mask, 0);
+
+#ifdef CONFIG_NETVM
+ if (!page && rx_emergency_get(PAGE_SIZE)) {
+ gfp_mask &= ~(__GFP_NOMEMALLOC | __GFP_NOWARN);
+ gfp_mask |= __GFP_MEMALLOC;
+ page = alloc_pages_node(node, gfp_mask, 0);
+ if (!page)
+ rx_emergency_put(PAGE_SIZE);
+ }
+#endif
+
return page;
}
+void __netdev_free_page(struct net_device *dev, struct page *page)
+{
+#ifdef CONFIG_NETVM
+ if (unlikely(page->reserve))
+ rx_emergency_put(PAGE_SIZE);
+#endif
+ __free_page(page);
+}
+
void skb_add_rx_frag(struct sk_buff *skb, int i, struct page *page, int off,
int size)
{
@@ -279,6 +330,34 @@ void skb_add_rx_frag(struct sk_buff *skb
skb->len += size;
skb->data_len += size;
skb->truesize += size;
+
+#ifdef CONFIG_NETVM
+ /*
+ * Fix-up the emergency accounting; make sure all pages match
+ * skb->emergency.
+ *
+ * This relies on page->reserve to be preserved between
+ * the call to __netdev_alloc_page() and this call.
+ */
+ if (skb_emergency(skb)) {
+ /*
+ * If the page was not an emergency alloc (ALLOC_NO_WATERMARK)
+ * we can use overcommit accounting, since we already have the
+ * memory.
+ */
+ if (!page->reserve)
+ rx_emergency_get_overcommit(PAGE_SIZE);
+ atomic_set(&page->frag_count, 1);
+ } else if (unlikely(page->reserve)) {
+ /*
+ * Rare case; the skb wasn't allocated under pressure but
+ * the page was. We need to return the page. This can offset
+ * the accounting a little, but its a constant shift, it does
+ * not accumulate.
+ */
+ rx_emergency_put(PAGE_SIZE);
+ }
+#endif
}
static void skb_drop_list(struct sk_buff **listp)
@@ -307,21 +386,45 @@ static void skb_clone_fraglist(struct sk
skb_get(list);
}
+static inline void skb_get_page(struct sk_buff *skb, struct page *page)
+{
+ get_page(page);
+ if (skb_emergency(skb))
+ atomic_inc(&page->frag_count);
+}
+
+static inline void skb_put_page(struct sk_buff *skb, struct page *page)
+{
+ if (skb_emergency(skb) && atomic_dec_and_test(&page->frag_count))
+ rx_emergency_put(PAGE_SIZE);
+ put_page(page);
+}
+
static void skb_release_data(struct sk_buff *skb)
{
if (!skb->cloned ||
!atomic_sub_return(skb->nohdr ? (1 << SKB_DATAREF_SHIFT) + 1 : 1,
&skb_shinfo(skb)->dataref)) {
+ int size;
+
+#ifdef NET_SKBUFF_DATA_USES_OFFSET
+ size = skb->end;
+#else
+ size = skb->end - skb->head;
+#endif
+
if (skb_shinfo(skb)->nr_frags) {
int i;
for (i = 0; i < skb_shinfo(skb)->nr_frags; i++)
- put_page(skb_shinfo(skb)->frags[i].page);
+ skb_put_page(skb, skb_shinfo(skb)->frags[i].page);
}
if (skb_shinfo(skb)->frag_list)
skb_drop_fraglist(skb);
kfree(skb->head);
+ if (skb_emergency(skb))
+ rx_emergency_put(size);
}
}
@@ -442,6 +545,7 @@ static void __copy_skb_header(struct sk_
#if defined(CONFIG_IP_VS) || defined(CONFIG_IP_VS_MODULE)
new->ipvs_property = old->ipvs_property;
#endif
+ new->emergency = old->emergency;
new->protocol = old->protocol;
new->mark = old->mark;
__nf_copy(new, old);
@@ -529,6 +633,9 @@ struct sk_buff *skb_clone(struct sk_buff
n->fclone = SKB_FCLONE_CLONE;
atomic_inc(fclone_ref);
} else {
+ if (skb_emergency(skb))
+ gfp_mask |= __GFP_MEMALLOC;
+
n = kmem_cache_alloc(skbuff_head_cache, gfp_mask);
if (!n)
return NULL;
@@ -560,6 +667,14 @@ static void copy_skb_header(struct sk_bu
skb_shinfo(new)->gso_type = skb_shinfo(old)->gso_type;
}
+static inline int skb_alloc_rx_flag(const struct sk_buff *skb)
+{
+ if (skb_emergency(skb))
+ return SKB_ALLOC_RX;
+
+ return 0;
+}
+
/**
* skb_copy - create private copy of an sk_buff
* @skb: buffer to copy
@@ -580,15 +695,17 @@ static void copy_skb_header(struct sk_bu
struct sk_buff *skb_copy(const struct sk_buff *skb, gfp_t gfp_mask)
{
int headerlen = skb->data - skb->head;
+ int size;
/*
* Allocate the copy buffer
*/
struct sk_buff *n;
#ifdef NET_SKBUFF_DATA_USES_OFFSET
- n = alloc_skb(skb->end + skb->data_len, gfp_mask);
+ size = skb->end + skb->data_len;
#else
- n = alloc_skb(skb->end - skb->head + skb->data_len, gfp_mask);
+ size = skb->end - skb->head + skb->data_len;
#endif
+ n = __alloc_skb(size, gfp_mask, skb_alloc_rx_flag(skb), -1);
if (!n)
return NULL;
@@ -623,12 +740,14 @@ struct sk_buff *pskb_copy(struct sk_buff
/*
* Allocate the copy buffer
*/
+ int size;
struct sk_buff *n;
#ifdef NET_SKBUFF_DATA_USES_OFFSET
- n = alloc_skb(skb->end, gfp_mask);
+ size = skb->end;
#else
- n = alloc_skb(skb->end - skb->head, gfp_mask);
+ size = skb->end - skb->head;
#endif
+ n = __alloc_skb(size, gfp_mask, skb_alloc_rx_flag(skb), -1);
if (!n)
goto out;
@@ -647,8 +766,9 @@ struct sk_buff *pskb_copy(struct sk_buff
int i;
for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
- skb_shinfo(n)->frags[i] = skb_shinfo(skb)->frags[i];
- get_page(skb_shinfo(n)->frags[i].page);
+ skb_frag_t *frag = &skb_shinfo(skb)->frags[i];
+ skb_shinfo(n)->frags[i] = *frag;
+ skb_get_page(n, frag->page);
}
skb_shinfo(n)->nr_frags = i;
}
@@ -696,6 +816,14 @@ int pskb_expand_head(struct sk_buff *skb
size = SKB_DATA_ALIGN(size);
+ if (skb_emergency(skb)) {
+ if (rx_emergency_get(size))
+ gfp_mask |= __GFP_MEMALLOC;
+ else
+ goto nodata;
+ } else
+ gfp_mask |= __GFP_NOMEMALLOC;
+
data = kmalloc(size + sizeof(struct skb_shared_info), gfp_mask);
if (!data)
goto nodata;
@@ -711,7 +839,7 @@ int pskb_expand_head(struct sk_buff *skb
sizeof(struct skb_shared_info));
for (i = 0; i < skb_shinfo(skb)->nr_frags; i++)
- get_page(skb_shinfo(skb)->frags[i].page);
+ skb_get_page(skb, skb_shinfo(skb)->frags[i].page);
if (skb_shinfo(skb)->frag_list)
skb_clone_fraglist(skb);
@@ -790,8 +918,8 @@ struct sk_buff *skb_copy_expand(const st
/*
* Allocate the copy buffer
*/
- struct sk_buff *n = alloc_skb(newheadroom + skb->len + newtailroom,
- gfp_mask);
+ struct sk_buff *n = __alloc_skb(newheadroom + skb->len + newtailroom,
+ gfp_mask, skb_alloc_rx_flag(skb), -1);
int oldheadroom = skb_headroom(skb);
int head_copy_len, head_copy_off;
int off;
@@ -908,7 +1036,7 @@ drop_pages:
skb_shinfo(skb)->nr_frags = i;
for (; i < nfrags; i++)
- put_page(skb_shinfo(skb)->frags[i].page);
+ skb_put_page(skb, skb_shinfo(skb)->frags[i].page);
if (skb_shinfo(skb)->frag_list)
skb_drop_fraglist(skb);
@@ -1077,7 +1205,7 @@ pull_pages:
k = 0;
for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
if (skb_shinfo(skb)->frags[i].size <= eat) {
- put_page(skb_shinfo(skb)->frags[i].page);
+ skb_put_page(skb, skb_shinfo(skb)->frags[i].page);
eat -= skb_shinfo(skb)->frags[i].size;
} else {
skb_shinfo(skb)->frags[k] = skb_shinfo(skb)->frags[i];
@@ -1849,6 +1977,7 @@ static inline void skb_split_no_header(s
skb_shinfo(skb1)->frags[k] = skb_shinfo(skb)->frags[i];
if (pos < len) {
+ struct page *page = skb_shinfo(skb)->frags[i].page;
/* Split frag.
* We have two variants in this case:
* 1. Move all the frag to the second
@@ -1857,7 +1986,7 @@ static inline void skb_split_no_header(s
* where splitting is expensive.
* 2. Split is accurately. We make this.
*/
- get_page(skb_shinfo(skb)->frags[i].page);
+ skb_get_page(skb1, page);
skb_shinfo(skb1)->frags[0].page_offset += len - pos;
skb_shinfo(skb1)->frags[0].size -= len - pos;
skb_shinfo(skb)->frags[i].size = len - pos;
@@ -2188,7 +2317,8 @@ struct sk_buff *skb_segment(struct sk_bu
if (hsize > len || !sg)
hsize = len;
- nskb = alloc_skb(hsize + doffset + headroom, GFP_ATOMIC);
+ nskb = __alloc_skb(hsize + doffset + headroom, GFP_ATOMIC,
+ skb_alloc_rx_flag(skb), -1);
if (unlikely(!nskb))
goto err;
@@ -2233,7 +2363,7 @@ struct sk_buff *skb_segment(struct sk_bu
BUG_ON(i >= nfrags);
*frag = skb_shinfo(skb)->frags[i];
- get_page(frag->page);
+ skb_get_page(nskb, frag->page);
size = frag->size;
if (pos < offset) {
@@ -2485,6 +2615,7 @@ EXPORT_SYMBOL(__pskb_pull_tail);
EXPORT_SYMBOL(__alloc_skb);
EXPORT_SYMBOL(__netdev_alloc_skb);
EXPORT_SYMBOL(__netdev_alloc_page);
+EXPORT_SYMBOL(__netdev_free_page);
EXPORT_SYMBOL(skb_add_rx_frag);
EXPORT_SYMBOL(pskb_copy);
EXPORT_SYMBOL(pskb_expand_head);
Index: linux-2.6/include/linux/mm_types.h
===================================================================
--- linux-2.6.orig/include/linux/mm_types.h
+++ linux-2.6/include/linux/mm_types.h
@@ -74,6 +74,7 @@ struct page {
pgoff_t index; /* Our offset within mapping. */
void *freelist; /* SLUB: freelist req. slab lock */
int reserve; /* page_alloc: page is a reserve page */
+ atomic_t frag_count; /* skb fragment use count */
};
struct list_head lru; /* Pageout list, eg. active_list
* protected by zone->lru_lock !
--
^ permalink raw reply
* [PATCH 16/29] netvm: INET reserves.
From: Peter Zijlstra @ 2007-12-14 15:39 UTC (permalink / raw)
To: Linus Torvalds, Andrew Morton, linux-kernel, linux-mm, netdev,
trond.myklebust
Cc: Peter Zijlstra
In-Reply-To: <20071214153907.770251000@chello.nl>
[-- Attachment #1: netvm-reserve-inet.patch --]
[-- Type: text/plain, Size: 12674 bytes --]
Add reserves for INET.
The two big users seem to be the route cache and ip-fragment cache.
Reserve the route cache under generic RX reserve, its usage is bounded by
the high reclaim watermark, and thus does not need further accounting.
Reserve the ip-fragement caches under SKB data reserve, these add to the
SKB RX limit. By ensuring we can at least receive as much data as fits in
the reassmbly line we avoid fragment attack deadlocks.
Use proc conv() routines to update these limits and return -ENOMEM to user
space.
Adds to the reserve tree:
total network reserve
network TX reserve
protocol TX pages
network RX reserve
+ IPv6 route cache
+ IPv4 route cache
SKB data reserve
+ IPv6 fragment cache
+ IPv4 fragment cache
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
---
net/ipv4/ip_fragment.c | 7 ++++
net/ipv4/route.c | 64 +++++++++++++++++++++++++++++++++++++++++++--
net/ipv4/sysctl_net_ipv4.c | 57 ++++++++++++++++++++++++++++++++++++++--
net/ipv6/reassembly.c | 7 ++++
net/ipv6/route.c | 64 +++++++++++++++++++++++++++++++++++++++++++--
net/ipv6/sysctl_net_ipv6.c | 57 ++++++++++++++++++++++++++++++++++++++--
6 files changed, 248 insertions(+), 8 deletions(-)
Index: linux-2.6/net/ipv4/sysctl_net_ipv4.c
===================================================================
--- linux-2.6.orig/net/ipv4/sysctl_net_ipv4.c
+++ linux-2.6/net/ipv4/sysctl_net_ipv4.c
@@ -21,6 +21,7 @@
#include <net/tcp.h>
#include <net/cipso_ipv4.h>
#include <net/inet_frag.h>
+#include <linux/reserve.h>
static int zero;
static int tcp_retr1_max = 255;
@@ -192,6 +193,57 @@ static int strategy_allowed_congestion_c
}
+static int ipv4_frag_bytes;
+extern struct mem_reserve ipv4_frag_reserve;
+
+static int proc_dointvec_fragment(struct ctl_table *table, int write,
+ struct file *filp, void __user *buffer, size_t *lenp,
+ loff_t *ppos)
+{
+ int old_bytes, ret;
+
+ if (!write)
+ ipv4_frag_bytes = ip4_frags_ctl.high_thresh;
+ old_bytes = ipv4_frag_bytes;
+
+ ret = proc_dointvec(table, write, filp, buffer, lenp, ppos);
+
+ if (!ret && write) {
+ ret = mem_reserve_kmalloc_set(&ipv4_frag_reserve, ipv4_frag_bytes);
+ if (!ret)
+ ip4_frags_ctl.high_thresh = ipv4_frag_bytes;
+ else
+ ipv4_frag_bytes = old_bytes;
+ }
+
+ return ret;
+}
+
+static int sysctl_intvec_fragment(struct ctl_table *table,
+ int __user *name, int nlen,
+ void __user *oldval, size_t __user *oldlenp,
+ void __user *newval, size_t newlen)
+{
+ int old_bytes, ret;
+ int write = (newval && newlen);
+
+ if (!write)
+ ipv4_frag_bytes = ip4_frags_ctl.high_thresh;
+ old_bytes = ipv4_frag_bytes;
+
+ ret = sysctl_intvec(table, name, nlen, oldval, oldlenp, newval, newlen);
+
+ if (!ret && write) {
+ ret = mem_reserve_kmalloc_set(&ipv4_frag_reserve, ipv4_frag_bytes);
+ if (!ret)
+ ip4_frags_ctl.high_thresh = ipv4_frag_bytes;
+ else
+ ipv4_frag_bytes = old_bytes;
+ }
+
+ return ret;
+}
+
static struct ctl_table ipv4_table[] = {
{
.ctl_name = NET_IPV4_TCP_TIMESTAMPS,
@@ -285,10 +337,11 @@ static struct ctl_table ipv4_table[] = {
{
.ctl_name = NET_IPV4_IPFRAG_HIGH_THRESH,
.procname = "ipfrag_high_thresh",
- .data = &ip4_frags_ctl.high_thresh,
+ .data = &ipv4_frag_bytes,
.maxlen = sizeof(int),
.mode = 0644,
- .proc_handler = &proc_dointvec
+ .proc_handler = &proc_dointvec_fragment,
+ .strategy = &sysctl_intvec_fragment,
},
{
.ctl_name = NET_IPV4_IPFRAG_LOW_THRESH,
Index: linux-2.6/net/ipv6/sysctl_net_ipv6.c
===================================================================
--- linux-2.6.orig/net/ipv6/sysctl_net_ipv6.c
+++ linux-2.6/net/ipv6/sysctl_net_ipv6.c
@@ -13,6 +13,58 @@
#include <net/ipv6.h>
#include <net/addrconf.h>
#include <net/inet_frag.h>
+#include <linux/reserve.h>
+
+static int ipv6_frag_bytes;
+extern struct mem_reserve ipv6_frag_reserve;
+
+static int proc_dointvec_fragment(struct ctl_table *table, int write,
+ struct file *filp, void __user *buffer, size_t *lenp,
+ loff_t *ppos)
+{
+ int old_bytes, ret;
+
+ if (!write)
+ ipv6_frag_bytes = ip6_frags_ctl.high_thresh;
+ old_bytes = ipv6_frag_bytes;
+
+ ret = proc_dointvec(table, write, filp, buffer, lenp, ppos);
+
+ if (!ret && write) {
+ ret = mem_reserve_kmalloc_set(&ipv6_frag_reserve, ipv6_frag_bytes);
+ if (!ret)
+ ip6_frags_ctl.high_thresh = ipv6_frag_bytes;
+ else
+ ipv6_frag_bytes = old_bytes;
+ }
+
+ return ret;
+}
+
+static int sysctl_intvec_fragment(struct ctl_table *table,
+ int __user *name, int nlen,
+ void __user *oldval, size_t __user *oldlenp,
+ void __user *newval, size_t newlen)
+{
+ int old_bytes, ret;
+ int write = (newval && newlen);
+
+ if (!write)
+ ipv6_frag_bytes = ip6_frags_ctl.high_thresh;
+ old_bytes = ipv6_frag_bytes;
+
+ ret = sysctl_intvec(table, name, nlen, oldval, oldlenp, newval, newlen);
+
+ if (!ret && write) {
+ ret = mem_reserve_kmalloc_set(&ipv6_frag_reserve, ipv6_frag_bytes);
+ if (!ret)
+ ip6_frags_ctl.high_thresh = ipv6_frag_bytes;
+ else
+ ipv6_frag_bytes = old_bytes;
+ }
+
+ return ret;
+}
static ctl_table ipv6_table[] = {
{
@@ -40,10 +92,11 @@ static ctl_table ipv6_table[] = {
{
.ctl_name = NET_IPV6_IP6FRAG_HIGH_THRESH,
.procname = "ip6frag_high_thresh",
- .data = &ip6_frags_ctl.high_thresh,
+ .data = &ipv6_frag_bytes,
.maxlen = sizeof(int),
.mode = 0644,
- .proc_handler = &proc_dointvec
+ .proc_handler = &proc_dointvec_fragment,
+ .strategy = &sysctl_intvec_fragment,
},
{
.ctl_name = NET_IPV6_IP6FRAG_LOW_THRESH,
Index: linux-2.6/net/ipv4/ip_fragment.c
===================================================================
--- linux-2.6.orig/net/ipv4/ip_fragment.c
+++ linux-2.6/net/ipv4/ip_fragment.c
@@ -44,6 +44,7 @@
#include <linux/udp.h>
#include <linux/inet.h>
#include <linux/netfilter_ipv4.h>
+#include <linux/reserve.h>
/* NOTE. Logic of IP defragmentation is parallel to corresponding IPv6
* code now. If you change something here, _PLEASE_ update ipv6/reassembly.c
@@ -607,6 +608,8 @@ int ip_defrag(struct sk_buff *skb, u32 u
return -ENOMEM;
}
+struct mem_reserve ipv4_frag_reserve;
+
void __init ipfrag_init(void)
{
ip4_frags.ctl = &ip4_frags_ctl;
@@ -618,6 +621,10 @@ void __init ipfrag_init(void)
ip4_frags.match = ip4_frag_match;
ip4_frags.frag_expire = ip_expire;
inet_frags_init(&ip4_frags);
+
+ mem_reserve_init(&ipv4_frag_reserve, "IPv4 fragment cache",
+ &net_skb_reserve);
+ mem_reserve_kmalloc_set(&ipv4_frag_reserve, ip4_frags_ctl.high_thresh);
}
EXPORT_SYMBOL(ip_defrag);
Index: linux-2.6/net/ipv6/reassembly.c
===================================================================
--- linux-2.6.orig/net/ipv6/reassembly.c
+++ linux-2.6/net/ipv6/reassembly.c
@@ -43,6 +43,7 @@
#include <linux/random.h>
#include <linux/jhash.h>
#include <linux/skbuff.h>
+#include <linux/reserve.h>
#include <net/sock.h>
#include <net/snmp.h>
@@ -632,6 +633,8 @@ static struct inet6_protocol frag_protoc
.flags = INET6_PROTO_NOPOLICY,
};
+struct mem_reserve ipv6_frag_reserve;
+
int __init ipv6_frag_init(void)
{
int ret;
@@ -650,6 +653,10 @@ int __init ipv6_frag_init(void)
inet_frags_init(&ip6_frags);
out:
return ret;
+
+ mem_reserve_init(&ipv6_frag_reserve, "IPv6 fragment cache",
+ &net_skb_reserve);
+ mem_reserve_kmalloc_set(&ipv6_frag_reserve, ip6_frags_ctl.high_thresh);
}
void ipv6_frag_exit(void)
Index: linux-2.6/net/ipv4/route.c
===================================================================
--- linux-2.6.orig/net/ipv4/route.c
+++ linux-2.6/net/ipv4/route.c
@@ -109,6 +109,7 @@
#ifdef CONFIG_SYSCTL
#include <linux/sysctl.h>
#endif
+#include <linux/reserve.h>
#define RT_FL_TOS(oldflp) \
((u32)(oldflp->fl4_tos & (IPTOS_RT_MASK | RTO_ONLINK)))
@@ -2815,6 +2816,59 @@ static int ipv4_sysctl_rtcache_flush_str
return 0;
}
+static int ipv4_route_size;
+static struct mem_reserve ipv4_route_reserve;
+
+static int proc_dointvec_route(struct ctl_table *table, int write,
+ struct file *filp, void __user *buffer, size_t *lenp,
+ loff_t *ppos)
+{
+ int old_size, ret;
+
+ if (!write)
+ ipv4_route_size = ip_rt_max_size;
+ old_size = ipv4_route_size;
+
+ ret = proc_dointvec(table, write, filp, buffer, lenp, ppos);
+
+ if (!ret && write) {
+ ret = mem_reserve_kmem_cache_set(&ipv4_route_reserve,
+ ipv4_dst_ops.kmem_cachep, ipv4_route_size);
+ if (!ret)
+ ip_rt_max_size = ipv4_route_size;
+ else
+ ipv4_route_size = old_size;
+ }
+
+ return ret;
+}
+
+static int sysctl_intvec_route(struct ctl_table *table,
+ int __user *name, int nlen,
+ void __user *oldval, size_t __user *oldlenp,
+ void __user *newval, size_t newlen)
+{
+ int old_size, ret;
+ int write = (newval && newlen);
+
+ if (!write)
+ ipv4_route_size = ip_rt_max_size;
+ old_size = ipv4_route_size;
+
+ ret = sysctl_intvec(table, name, nlen, oldval, oldlenp, newval, newlen);
+
+ if (!ret && write) {
+ ret = mem_reserve_kmem_cache_set(&ipv4_route_reserve,
+ ipv4_dst_ops.kmem_cachep, ipv4_route_size);
+ if (!ret)
+ ip_rt_max_size = ipv4_route_size;
+ else
+ ipv4_route_size = old_size;
+ }
+
+ return ret;
+}
+
ctl_table ipv4_route_table[] = {
{
.ctl_name = NET_IPV4_ROUTE_FLUSH,
@@ -2854,10 +2908,11 @@ ctl_table ipv4_route_table[] = {
{
.ctl_name = NET_IPV4_ROUTE_MAX_SIZE,
.procname = "max_size",
- .data = &ip_rt_max_size,
+ .data = &ipv4_route_size,
.maxlen = sizeof(int),
.mode = 0644,
- .proc_handler = &proc_dointvec,
+ .proc_handler = &proc_dointvec_route,
+ .strategy = &sysctl_intvec_route,
},
{
/* Deprecated. Use gc_min_interval_ms */
@@ -3032,6 +3087,11 @@ int __init ip_rt_init(void)
ipv4_dst_ops.gc_thresh = (rt_hash_mask + 1);
ip_rt_max_size = (rt_hash_mask + 1) * 16;
+ mem_reserve_init(&ipv4_route_reserve, "IPv4 route cache",
+ &net_rx_reserve);
+ mem_reserve_kmem_cache_set(&ipv4_route_reserve,
+ ipv4_dst_ops.kmem_cachep, ip_rt_max_size);
+
devinet_init();
ip_fib_init();
Index: linux-2.6/net/ipv6/route.c
===================================================================
--- linux-2.6.orig/net/ipv6/route.c
+++ linux-2.6/net/ipv6/route.c
@@ -38,6 +38,7 @@
#include <linux/in6.h>
#include <linux/init.h>
#include <linux/if_arp.h>
+#include <linux/reserve.h>
#include <linux/proc_fs.h>
#include <linux/seq_file.h>
#include <net/net_namespace.h>
@@ -2405,6 +2406,59 @@ int ipv6_sysctl_rtcache_flush(ctl_table
return -EINVAL;
}
+static int ipv6_route_size;
+static struct mem_reserve ipv6_route_reserve;
+
+static int proc_dointvec_route(struct ctl_table *table, int write,
+ struct file *filp, void __user *buffer, size_t *lenp,
+ loff_t *ppos)
+{
+ int old_size, ret;
+
+ if (!write)
+ ipv6_route_size = ip6_rt_max_size;
+ old_size = ipv6_route_size;
+
+ ret = proc_dointvec(table, write, filp, buffer, lenp, ppos);
+
+ if (!ret && write) {
+ ret = mem_reserve_kmem_cache_set(&ipv6_route_reserve,
+ ip6_dst_ops.kmem_cachep, ipv6_route_size);
+ if (!ret)
+ ip6_rt_max_size = ipv6_route_size;
+ else
+ ipv6_route_size = old_size;
+ }
+
+ return ret;
+}
+
+static int sysctl_intvec_route(struct ctl_table *table,
+ int __user *name, int nlen,
+ void __user *oldval, size_t __user *oldlenp,
+ void __user *newval, size_t newlen)
+{
+ int old_size, ret;
+ int write = (newval && newlen);
+
+ if (!write)
+ ipv6_route_size = ip6_rt_max_size;
+ old_size = ipv6_route_size;
+
+ ret = sysctl_intvec(table, name, nlen, oldval, oldlenp, newval, newlen);
+
+ if (!ret && write) {
+ ret = mem_reserve_kmem_cache_set(&ipv6_route_reserve,
+ ip6_dst_ops.kmem_cachep, ipv6_route_size);
+ if (!ret)
+ ip6_rt_max_size = ipv6_route_size;
+ else
+ ipv6_route_size = old_size;
+ }
+
+ return ret;
+}
+
ctl_table ipv6_route_table[] = {
{
.procname = "flush",
@@ -2424,10 +2478,11 @@ ctl_table ipv6_route_table[] = {
{
.ctl_name = NET_IPV6_ROUTE_MAX_SIZE,
.procname = "max_size",
- .data = &ip6_rt_max_size,
+ .data = &ipv6_route_size,
.maxlen = sizeof(int),
.mode = 0644,
- .proc_handler = &proc_dointvec,
+ .proc_handler = &proc_dointvec_route,
+ .strategy = &sysctl_intvec_route,
},
{
.ctl_name = NET_IPV6_ROUTE_GC_MIN_INTERVAL,
@@ -2509,6 +2564,11 @@ int __init ip6_route_init(void)
ip6_dst_blackhole_ops.kmem_cachep = ip6_dst_ops.kmem_cachep;
+ mem_reserve_init(&ipv6_route_reserve, "IPv6 route cache",
+ &net_rx_reserve);
+ mem_reserve_kmem_cache_set(&ipv6_route_reserve,
+ ip6_dst_ops.kmem_cachep, ip6_rt_max_size);
+
ret = fib6_init();
if (ret)
goto out_kmem_cache;
--
^ permalink raw reply
* [PATCH 13/29] net: packet split receive api
From: Peter Zijlstra @ 2007-12-14 15:39 UTC (permalink / raw)
To: Linus Torvalds, Andrew Morton, linux-kernel, linux-mm, netdev,
trond.myklebust
Cc: Peter Zijlstra
In-Reply-To: <20071214153907.770251000@chello.nl>
[-- Attachment #1: net-ps_rx.patch --]
[-- Type: text/plain, Size: 5645 bytes --]
Add some packet-split receive hooks.
For one this allows to do NUMA node affine page allocs. Later on these hooks
will be extended to do emergency reserve allocations for fragments.
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
---
drivers/net/e1000/e1000_main.c | 8 ++------
drivers/net/sky2.c | 16 ++++++----------
include/linux/skbuff.h | 23 +++++++++++++++++++++++
net/core/skbuff.c | 20 ++++++++++++++++++++
4 files changed, 51 insertions(+), 16 deletions(-)
Index: linux-2.6/drivers/net/e1000/e1000_main.c
===================================================================
--- linux-2.6.orig/drivers/net/e1000/e1000_main.c
+++ linux-2.6/drivers/net/e1000/e1000_main.c
@@ -4392,12 +4392,8 @@ e1000_clean_rx_irq_ps(struct e1000_adapt
pci_unmap_page(pdev, ps_page_dma->ps_page_dma[j],
PAGE_SIZE, PCI_DMA_FROMDEVICE);
ps_page_dma->ps_page_dma[j] = 0;
- skb_fill_page_desc(skb, j, ps_page->ps_page[j], 0,
- length);
+ skb_add_rx_frag(skb, j, ps_page->ps_page[j], 0, length);
ps_page->ps_page[j] = NULL;
- skb->len += length;
- skb->data_len += length;
- skb->truesize += length;
}
/* strip the ethernet crc, problem is we're using pages now so
@@ -4605,7 +4601,7 @@ e1000_alloc_rx_buffers_ps(struct e1000_a
if (j < adapter->rx_ps_pages) {
if (likely(!ps_page->ps_page[j])) {
ps_page->ps_page[j] =
- alloc_page(GFP_ATOMIC);
+ netdev_alloc_page(netdev);
if (unlikely(!ps_page->ps_page[j])) {
adapter->alloc_rx_buff_failed++;
goto no_buffers;
Index: linux-2.6/include/linux/skbuff.h
===================================================================
--- linux-2.6.orig/include/linux/skbuff.h
+++ linux-2.6/include/linux/skbuff.h
@@ -851,6 +851,9 @@ static inline void skb_fill_page_desc(st
skb_shinfo(skb)->nr_frags = i + 1;
}
+extern void skb_add_rx_frag(struct sk_buff *skb, int i, struct page *page,
+ int off, int size);
+
#define SKB_PAGE_ASSERT(skb) BUG_ON(skb_shinfo(skb)->nr_frags)
#define SKB_FRAG_ASSERT(skb) BUG_ON(skb_shinfo(skb)->frag_list)
#define SKB_LINEAR_ASSERT(skb) BUG_ON(skb_is_nonlinear(skb))
@@ -1344,6 +1347,26 @@ static inline struct sk_buff *netdev_all
return __netdev_alloc_skb(dev, length, GFP_ATOMIC);
}
+extern struct page *__netdev_alloc_page(struct net_device *dev, gfp_t gfp_mask);
+
+/**
+ * netdev_alloc_page - allocate a page for ps-rx on a specific device
+ * @dev: network device to receive on
+ *
+ * Allocate a new page node local to the specified device.
+ *
+ * %NULL is returned if there is no free memory.
+ */
+static inline struct page *netdev_alloc_page(struct net_device *dev)
+{
+ return __netdev_alloc_page(dev, GFP_ATOMIC);
+}
+
+static inline void netdev_free_page(struct net_device *dev, struct page *page)
+{
+ __free_page(page);
+}
+
/**
* skb_clone_writable - is the header of a clone writable
* @skb: buffer to check
Index: linux-2.6/net/core/skbuff.c
===================================================================
--- linux-2.6.orig/net/core/skbuff.c
+++ linux-2.6/net/core/skbuff.c
@@ -263,6 +263,24 @@ struct sk_buff *__netdev_alloc_skb(struc
return skb;
}
+struct page *__netdev_alloc_page(struct net_device *dev, gfp_t gfp_mask)
+{
+ int node = dev->dev.parent ? dev_to_node(dev->dev.parent) : -1;
+ struct page *page;
+
+ page = alloc_pages_node(node, gfp_mask, 0);
+ return page;
+}
+
+void skb_add_rx_frag(struct sk_buff *skb, int i, struct page *page, int off,
+ int size)
+{
+ skb_fill_page_desc(skb, i, page, off, size);
+ skb->len += size;
+ skb->data_len += size;
+ skb->truesize += size;
+}
+
static void skb_drop_list(struct sk_buff **listp)
{
struct sk_buff *list = *listp;
@@ -2466,6 +2484,8 @@ EXPORT_SYMBOL(kfree_skb);
EXPORT_SYMBOL(__pskb_pull_tail);
EXPORT_SYMBOL(__alloc_skb);
EXPORT_SYMBOL(__netdev_alloc_skb);
+EXPORT_SYMBOL(__netdev_alloc_page);
+EXPORT_SYMBOL(skb_add_rx_frag);
EXPORT_SYMBOL(pskb_copy);
EXPORT_SYMBOL(pskb_expand_head);
EXPORT_SYMBOL(skb_checksum);
Index: linux-2.6/drivers/net/sky2.c
===================================================================
--- linux-2.6.orig/drivers/net/sky2.c
+++ linux-2.6/drivers/net/sky2.c
@@ -1198,7 +1198,7 @@ static struct sk_buff *sky2_rx_alloc(str
}
for (i = 0; i < sky2->rx_nfrags; i++) {
- struct page *page = alloc_page(GFP_ATOMIC);
+ struct page *page = netdev_alloc_page(sky2->netdev);
if (!page)
goto free_partial;
@@ -2072,8 +2072,8 @@ static struct sk_buff *receive_copy(stru
}
/* Adjust length of skb with fragments to match received data */
-static void skb_put_frags(struct sk_buff *skb, unsigned int hdr_space,
- unsigned int length)
+static void skb_put_frags(struct sky2_port *sky2, struct sk_buff *skb,
+ unsigned int hdr_space, unsigned int length)
{
int i, num_frags;
unsigned int size;
@@ -2090,15 +2090,11 @@ static void skb_put_frags(struct sk_buff
if (length == 0) {
/* don't need this page */
- __free_page(frag->page);
+ netdev_free_page(sky2->netdev, frag->page);
--skb_shinfo(skb)->nr_frags;
} else {
size = min(length, (unsigned) PAGE_SIZE);
-
- frag->size = size;
- skb->data_len += size;
- skb->truesize += size;
- skb->len += size;
+ skb_add_rx_frag(skb, i, frag->page, 0, size);
length -= size;
}
}
@@ -2125,7 +2121,7 @@ static struct sk_buff *receive_new(struc
sky2_rx_map_skb(sky2->hw->pdev, re, hdr_space);
if (skb_shinfo(skb)->nr_frags)
- skb_put_frags(skb, hdr_space, length);
+ skb_put_frags(sky2, skb, hdr_space, length);
else
skb_put(skb, length);
return skb;
--
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox