* [GIT PULL] Fix bugs in Slow-work, FS-Cache and CacheFiles
@ 2009-11-20 10:48 David Howells
2009-11-20 23:01 ` [GIT PULL v2] " David Howells
0 siblings, 1 reply; 4+ messages in thread
From: David Howells @ 2009-11-20 10:48 UTC (permalink / raw)
To: torvalds, akpm
Cc: cluster-devel, nfsv4, linux-kernel, linux-cachefs, jens.axboe,
linux-fsdevel, linux-cifs-client
Hi Linus,
Could you pull the attached patches before 2.6.32 please? For the most part
they are bug fixes for the slow-work, FS-Cache and CacheFiles facilities, with
a few patches to produce better debugging information, and a couple of patches
to slow work from Jens that all the rest are built on top of.
These vastly improve the stability of FS-Cache under extremely high load -
especially when the system is running out of memory regularly. I wrote a
better test case:
http://people.redhat.com/~dhowells/slurp.c
which helped enormously in reproducing these bugs. Testing has involved
running repeated invocations of slurp over five NFS4 mounted directory trees
totalling ~40GB in N file spread over 100,000 dirs on a machine with 1GB of
RAM and 4GB of disk cache space. Note that slurp, being massively parallel,
will abandon a part of the tree if it hits any error (such as ENOMEM) - so not
all the data set will necessarily be used in any particular run.
The patches can also be downloaded as:
http://people.redhat.com/~dhowells/fscache/patches/fscache-fixes.tar.bz2
The changes don't affect very much outside of FS-Cache and its dependents (NFS,
AFS and 9P), but the slow-work changes do affect CIFS and GFS2.
Thanks,
David
---
The following changes since commit 66b00a7c93ec782d118d2c03bd599cfd041e80a1:
Linus Torvalds (1):
Merge branch 'fixes' of git://git.kernel.org/.../davej/cpufreq
are available in the git repository at:
git://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-2.6-fscache.git master
David Howells (25):
SLOW_WORK: Wait for outstanding work items belonging to a module to clear
SLOW_WORK: Allow the work items to be viewed through a /proc file
SLOW_WORK: Allow the owner of a work item to determine if it is queued or not
SLOW_WORK: Allow a requeueable work item to sleep till the thread is needed
FS-Cache: Annotate slow-work runqueue proc lines for FS-Cache work items
FS-Cache: Allow the current state of all objects to be dumped
FS-Cache: Add counters for entry/exit to/from cache operation functions
FS-Cache: Clear netfs pointers in cookie after detaching object, not before
FS-Cache: Use radix tree preload correctly in tracking of pages to be stored
FS-Cache: Permit cache retrieval ops to be interrupted in the initial wait phase
FS-Cache: The object-available state can't rely on the cookie to be available
FS-Cache: Fix lock misorder in fscache_write_op()
FS-Cache: Don't delete pending pages from the page-store tracking tree
FS-Cache: Handle read request vs lookup, creation or other cache failure
FS-Cache: Handle pages pending storage that get evicted under OOM conditions
FS-Cache: Add a retirement stat counter
FS-Cache: Make sure FSCACHE_COOKIE_LOOKING_UP cleared on lookup failure
FS-Cache: Start processing an object's operations on that object's death
FS-Cache: Actually requeue an object when requested
CacheFiles: Don't write a full page if there's only a partial page to cache
CacheFiles: Handle truncate unlocking the page we're reading
CacheFiles: Mark parent directory locks as I_MUTEX_PARENT to keep lockdep happy
CacheFiles: Better showing of debugging information in active object problems
CacheFiles: Catch an overly long wait for an old active object
CacheFiles: Don't log lookup/create failing with ENOBUFS
Jens Axboe (3):
SLOW_WORK: Make slow_work_ops ->get_ref/->put_ref optional
SLOW_WORK: Add support for cancellation of slow work
SLOW_WORK: Add delayed_slow_work support
Documentation/filesystems/caching/fscache.txt | 110 +++++
Documentation/filesystems/caching/netfs-api.txt | 21 +-
Documentation/slow-work.txt | 160 +++++++-
fs/9p/cache.c | 14 +-
fs/afs/file.c | 15 +-
fs/cachefiles/interface.c | 32 ++-
fs/cachefiles/namei.c | 187 +++++++--
fs/cachefiles/rdwr.c | 128 ++++++-
fs/fscache/Kconfig | 7 +
fs/fscache/Makefile | 1 +
fs/fscache/cache.c | 5 +
fs/fscache/cookie.c | 26 +-
fs/fscache/internal.h | 55 +++
fs/fscache/main.c | 6 +-
fs/fscache/object-list.c | 432 ++++++++++++++++++++
fs/fscache/object.c | 104 +++++-
fs/fscache/operation.c | 120 +++++--
fs/fscache/page.c | 273 ++++++++++---
fs/fscache/proc.c | 13 +
fs/fscache/stats.c | 94 ++++-
fs/gfs2/main.c | 4 +-
fs/gfs2/recovery.c | 1 +
fs/nfs/fscache.c | 10 +-
include/linux/fscache-cache.h | 40 ++-
include/linux/fscache.h | 27 ++
include/linux/slow-work.h | 72 ++++-
init/Kconfig | 10 +
kernel/Makefile | 1 +
kernel/slow-work-proc.c | 227 +++++++++++
kernel/slow-work.c | 494 +++++++++++++++++++++--
kernel/slow-work.h | 72 ++++
lib/radix-tree.c | 5 +-
32 files changed, 2502 insertions(+), 264 deletions(-)
create mode 100644 fs/fscache/object-list.c
create mode 100644 kernel/slow-work-proc.c
create mode 100644 kernel/slow-work.h
^ permalink raw reply [flat|nested] 4+ messages in thread
* [GIT PULL v2] Fix bugs in Slow-work, FS-Cache and CacheFiles
2009-11-20 10:48 [GIT PULL] Fix bugs in Slow-work, FS-Cache and CacheFiles David Howells
@ 2009-11-20 23:01 ` David Howells
2009-12-01 6:56 ` [PATCH] slow-work: Fix build bug in the !CONFIG_MODULES case Ingo Molnar
0 siblings, 1 reply; 4+ messages in thread
From: David Howells @ 2009-11-20 23:01 UTC (permalink / raw)
To: torvalds, akpm
Cc: cluster-devel, nfsv4, linux-kernel, linux-cachefs, jens.axboe,
linux-fsdevel, linux-cifs-client
Hi Linus,
Could you pull the attached patches before 2.6.32 please? I've added three
one-liner patches to fix some silly bugs.
For the most part these patches are bug fixes for the slow-work, FS-Cache and
CacheFiles facilities, with a few patches to produce better debugging
information, and a couple of patches to slow work from Jens that all the rest
are built on top of.
These vastly improve the stability of FS-Cache under extremely high load -
especially when the system is running out of memory regularly. I wrote a
better test case:
http://people.redhat.com/~dhowells/slurp.c
which helped enormously in reproducing these bugs. Testing has involved
running repeated invocations of slurp over five NFS4 mounted directory trees
totalling ~40GB in N file spread over 100,000 dirs on a machine with 1GB of
RAM and 4GB of disk cache space. Note that slurp, being massively parallel,
will abandon a part of the tree if it hits any error (such as ENOMEM) - so not
all the data set will necessarily be used in any particular run.
The patches can also be downloaded as:
http://people.redhat.com/~dhowells/fscache/patches/fscache-fixes.tar.bz2
The changes don't affect very much outside of FS-Cache and its dependents (NFS,
AFS and 9P), but the slow-work changes do affect CIFS and GFS2.
Thanks,
David
---
The following changes since commit 66b00a7c93ec782d118d2c03bd599cfd041e80a1:
Linus Torvalds (1):
Merge branch 'fixes' of git://git.kernel.org/.../davej/cpufreq
are available in the git repository at:
git://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-2.6-fscache.git master
David Howells (28):
SLOW_WORK: Wait for outstanding work items belonging to a module to clear
SLOW_WORK: Allow the work items to be viewed through a /proc file
SLOW_WORK: Allow the owner of a work item to determine if it is queued or not
SLOW_WORK: Allow a requeueable work item to sleep till the thread is needed
FS-Cache: Annotate slow-work runqueue proc lines for FS-Cache work items
FS-Cache: Allow the current state of all objects to be dumped
FS-Cache: Add counters for entry/exit to/from cache operation functions
FS-Cache: Clear netfs pointers in cookie after detaching object, not before
FS-Cache: Use radix tree preload correctly in tracking of pages to be stored
FS-Cache: Permit cache retrieval ops to be interrupted in the initial wait phase
FS-Cache: The object-available state can't rely on the cookie to be available
FS-Cache: Fix lock misorder in fscache_write_op()
FS-Cache: Don't delete pending pages from the page-store tracking tree
FS-Cache: Handle read request vs lookup, creation or other cache failure
FS-Cache: Handle pages pending storage that get evicted under OOM conditions
FS-Cache: Add a retirement stat counter
FS-Cache: Make sure FSCACHE_COOKIE_LOOKING_UP cleared on lookup failure
FS-Cache: Start processing an object's operations on that object's death
FS-Cache: Actually requeue an object when requested
CacheFiles: Don't write a full page if there's only a partial page to cache
CacheFiles: Handle truncate unlocking the page we're reading
CacheFiles: Mark parent directory locks as I_MUTEX_PARENT to keep lockdep happy
CacheFiles: Better showing of debugging information in active object problems
CacheFiles: Catch an overly long wait for an old active object
CacheFiles: Don't log lookup/create failing with ENOBUFS
SLOW_WORK: Fix CIFS to pass THIS_MODULE to slow_work_register_user()
SLOW_WORK: Fix GFS2 to #include <linux/module.h> before using THIS_MODULE
FS-Cache: Provide nop fscache_stat_d() if CONFIG_FSCACHE_STATS=n
Jens Axboe (3):
SLOW_WORK: Make slow_work_ops ->get_ref/->put_ref optional
SLOW_WORK: Add support for cancellation of slow work
SLOW_WORK: Add delayed_slow_work support
Documentation/filesystems/caching/fscache.txt | 110 +++++
Documentation/filesystems/caching/netfs-api.txt | 21 +-
Documentation/slow-work.txt | 160 +++++++-
fs/9p/cache.c | 14 +-
fs/afs/file.c | 15 +-
fs/cachefiles/interface.c | 32 ++-
fs/cachefiles/namei.c | 187 +++++++--
fs/cachefiles/rdwr.c | 128 ++++++-
fs/cifs/cifsfs.c | 2 +-
fs/fscache/Kconfig | 7 +
fs/fscache/Makefile | 1 +
fs/fscache/cache.c | 5 +
fs/fscache/cookie.c | 26 +-
fs/fscache/internal.h | 56 +++
fs/fscache/main.c | 6 +-
fs/fscache/object-list.c | 432 ++++++++++++++++++++
fs/fscache/object.c | 104 +++++-
fs/fscache/operation.c | 120 +++++--
fs/fscache/page.c | 273 ++++++++++---
fs/fscache/proc.c | 13 +
fs/fscache/stats.c | 94 ++++-
fs/gfs2/main.c | 4 +-
fs/gfs2/recovery.c | 2 +
fs/nfs/fscache.c | 10 +-
include/linux/fscache-cache.h | 40 ++-
include/linux/fscache.h | 27 ++
include/linux/slow-work.h | 72 ++++-
init/Kconfig | 10 +
kernel/Makefile | 1 +
kernel/slow-work-proc.c | 227 +++++++++++
kernel/slow-work.c | 494 +++++++++++++++++++++--
kernel/slow-work.h | 72 ++++
lib/radix-tree.c | 5 +-
33 files changed, 2505 insertions(+), 265 deletions(-)
create mode 100644 fs/fscache/object-list.c
create mode 100644 kernel/slow-work-proc.c
create mode 100644 kernel/slow-work.h
^ permalink raw reply [flat|nested] 4+ messages in thread
* [PATCH] slow-work: Fix build bug in the !CONFIG_MODULES case
2009-11-20 23:01 ` [GIT PULL v2] " David Howells
@ 2009-12-01 6:56 ` Ingo Molnar
2009-12-01 7:32 ` Stephen Rothwell
0 siblings, 1 reply; 4+ messages in thread
From: Ingo Molnar @ 2009-12-01 6:56 UTC (permalink / raw)
To: David Howells
Cc: torvalds, akpm, steved, jens.axboe, linux-cachefs, nfsv4,
linux-fsdevel, cluster-devel, linux-kernel, linux-cifs-client
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=unknown-8bit, Size: 3778 bytes --]
* David Howells <dhowells@redhat.com> wrote:
> Hi Linus,
>
> Could you pull the attached patches before 2.6.32 please? I've added three
> one-liner patches to fix some silly bugs.
>
> For the most part these patches are bug fixes for the slow-work, FS-Cache and
> CacheFiles facilities, with a few patches to produce better debugging
> information, and a couple of patches to slow work from Jens that all the rest
> are built on top of.
these slow-work changes have introduced a new !CONFIG_MODULES build bug
on x86 (and all other architectures i suspect):
kernel/slow-work.c: In function ‘slow_work_execute’:
kernel/slow-work.c:313: error: ‘slow_work_thread_processing’ undeclared (first use in this function)
kernel/slow-work.c:313: error: (Each undeclared identifier is reported only once
kernel/slow-work.c:313: error: for each function it appears in.)
kernel/slow-work.c: In function ‘slow_work_wait_for_items’:
kernel/slow-work.c:950: error: ‘slow_work_unreg_sync_lock’ undeclared (first use in this function)
kernel/slow-work.c:951: error: ‘slow_work_unreg_wq’ undeclared (first use in this function)
kernel/slow-work.c:961: error: ‘slow_work_unreg_work_item’ undeclared (first use in this function)
kernel/slow-work.c:974: error: ‘slow_work_unreg_module’ undeclared (first use in this function)
kernel/slow-work.c:977: error: ‘slow_work_thread_processing’ undeclared (first use in this function)
Commit 3bde31a ("SLOW_WORK: Allow a requeueable work item to sleep till
the thread is needed") broke the !CONFIG_MODULES build by relying on
variables that are only available (and only make sense) with module
support enabled. Fix it.
Note, to fix it in a minimal way for .32 i've simply extended the
existing #ifdef's. In v2.6.33 a cleaner change would be to move the
whole wait-for-modules block into a separate slow-work-wait-modules.c
file or so - that will allow the removal of 6 #ifdefs.
Thanks,
Ingo
--------------------------->
From ac73995e9be78ba5dcf3db843d7e04295decabcd Mon Sep 17 00:00:00 2001
From: Ingo Molnar <mingo@elte.hu>
Date: Tue, 1 Dec 2009 07:47:52 +0100
Subject: [PATCH] slow-work: Fix build bug in the !CONFIG_MODULES case
Commit 3bde31a ("SLOW_WORK: Allow a requeueable work item to sleep till
the thread is needed") broke the !CONFIG_MODULES build by relying on
variables that are only available (and only make sense) with module
support enabled. Fix it.
Cc: David Howells <dhowells@redhat.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
---
kernel/slow-work.c | 4 ++++
1 files changed, 4 insertions(+), 0 deletions(-)
diff --git a/kernel/slow-work.c b/kernel/slow-work.c
index da94f3c..5bae807 100644
--- a/kernel/slow-work.c
+++ b/kernel/slow-work.c
@@ -310,7 +310,9 @@ auto_requeue:
else
list_add_tail(&work->link, &slow_work_queue);
spin_unlock_irq(&slow_work_queue_lock);
+#ifdef CONFIG_MODULES
slow_work_thread_processing[id] = NULL;
+#endif
return true;
}
@@ -943,6 +945,7 @@ EXPORT_SYMBOL(slow_work_register_user);
*/
static void slow_work_wait_for_items(struct module *module)
{
+#ifdef CONFIG_MODULES
DECLARE_WAITQUEUE(myself, current);
struct slow_work *work;
int loop;
@@ -989,6 +992,7 @@ static void slow_work_wait_for_items(struct module *module)
remove_wait_queue(&slow_work_unreg_wq, &myself);
mutex_unlock(&slow_work_unreg_sync_lock);
+#endif
}
/**
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [PATCH] slow-work: Fix build bug in the !CONFIG_MODULES case
2009-12-01 6:56 ` [PATCH] slow-work: Fix build bug in the !CONFIG_MODULES case Ingo Molnar
@ 2009-12-01 7:32 ` Stephen Rothwell
0 siblings, 0 replies; 4+ messages in thread
From: Stephen Rothwell @ 2009-12-01 7:32 UTC (permalink / raw)
To: Ingo Molnar
Cc: David Howells, torvalds, akpm, steved, jens.axboe, linux-cachefs,
nfsv4, linux-fsdevel, cluster-devel, linux-kernel,
linux-cifs-client, linux-next
[-- Attachment #1: Type: text/plain, Size: 1063 bytes --]
Hi Ingo,
On Tue, 1 Dec 2009 07:56:05 +0100 Ingo Molnar <mingo@elte.hu> wrote:
>
> these slow-work changes have introduced a new !CONFIG_MODULES build bug
> on x86 (and all other architectures i suspect):
Thanks for the heads up!
> From: Ingo Molnar <mingo@elte.hu>
> Date: Tue, 1 Dec 2009 07:47:52 +0100
> Subject: [PATCH] slow-work: Fix build bug in the !CONFIG_MODULES case
>
> Commit 3bde31a ("SLOW_WORK: Allow a requeueable work item to sleep till
> the thread is needed") broke the !CONFIG_MODULES build by relying on
> variables that are only available (and only make sense) with module
> support enabled. Fix it.
>
> Cc: David Howells <dhowells@redhat.com>
> Cc: Linus Torvalds <torvalds@linux-foundation.org>
> Cc: Andrew Morton <akpm@linux-foundation.org>
> Signed-off-by: Ingo Molnar <mingo@elte.hu>
I have applied this to linux-next for today, just so people don't waste
their time finding this breakage again.
--
Cheers,
Stephen Rothwell sfr@canb.auug.org.au
http://www.canb.auug.org.au/~sfr/
[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2009-12-01 7:32 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2009-11-20 10:48 [GIT PULL] Fix bugs in Slow-work, FS-Cache and CacheFiles David Howells
2009-11-20 23:01 ` [GIT PULL v2] " David Howells
2009-12-01 6:56 ` [PATCH] slow-work: Fix build bug in the !CONFIG_MODULES case Ingo Molnar
2009-12-01 7:32 ` Stephen Rothwell
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).