Re: [PATCH 00/11] writeback bug fixes and simplifications

Re: [PATCH 00/11] writeback bug fixes and simplifications

[Sascha Warner]

Hi,

I applied your patches to 2.6.24-rc6-mm1, but now I am faced with one
pdflush often using 100% CPU for a long time. There seem to be some rare
pauses from its 100% usage, however.

On ~23 minutes uptime i have ~19 minutes pdflush runtime.

This is on E6600, x86_64, 2 Gig RAM, SATA HDD, running on gentoo ~x64_64

Let me know if you need more info.


Prost,
Sascha Warner

Re: [PATCH 00/11] writeback bug fixes and simplifications

[Sascha Warner]

Andrew Morton wrote:
> On Thu, 27 Dec 2007 23:08:40 +0100 Sascha Warner <prx@cinatas.ath.cx> wrote:
>
>   
>> Hi,
>>
>> I applied your patches to 2.6.24-rc6-mm1, but now I am faced with one
>> pdflush often using 100% CPU for a long time. There seem to be some rare
>> pauses from its 100% usage, however.
>>
>> On ~23 minutes uptime i have ~19 minutes pdflush runtime.
>>
>> This is on E6600, x86_64, 2 Gig RAM, SATA HDD, running on gentoo ~x64_64
>>
>> Let me know if you need more info.
>>
>>     
>
> (some) cc's restored.  Please, always do reply-to-all.
>   
Hi Wu,

Your latest patch does fix the seen pdflush 100% issue :)

P.S.: Andrew, sorry my subject pretended I was subscribed to lkml, I am not.
I hoped for the script at lkml to pull the threads together because of
the subject.

Thank you,
Sascha Warner

[PATCH 00/11] writeback bug fixes and simplifications

[Fengguang Wu]

Andrew,

Here are some writeback bug fixes and code cleanups, based on 2.6.24-rc6-mm1.

The detailed description can be found in the first patch, which introduces the
new facility for the following patches to use.

The first two patches combined should fix the "slow writes for concurrent large
and small file writes" bug which Michael Rubin tries to fix.

[PATCH 01/11] writeback: introduce s_more_io_wait for inodes to be re-synced af
[PATCH 02/11] writeback: requeue_io_wait() on blocked kupdate
[PATCH 03/11] writeback: replace redirty_tail() on more-pages-to-sync
[PATCH 04/11] writeback: requeue_io_wait() on locked inode
[PATCH 05/11] writeback: requeue_io_wait() on congested blockdev
[PATCH 06/11] writeback: requeue_io_wait() on locked buffers
[PATCH 07/11] writeback: requeue_io() on redirtied inode

More code cleanups:

[PATCH 08/11] writeback: introduce queue_dirty()
[PATCH 09/11] writeback: replace redirty_tail() on memory-backed bdi
[PATCH 10/11] writeback: remove redirty_tail()
[PATCH 11/11] writeback: __sync_single_inode() code cleanup


Diffstat:

 fs/fs-writeback.c         |  125 +++++++++++++-----------------------
 fs/super.c                |    1
 include/linux/fs.h        |    1
 include/linux/writeback.h |    1
 mm/page-writeback.c       |   56 +++++++++-------
 5 files changed, 82 insertions(+), 102 deletions(-)

Thank you,
Fengguang
--

[PATCH 01/11] writeback: introduce s_more_io_wait for inodes to be re-synced after a while

[Fengguang Wu]

[-- Attachment #1: writeback-more_io_wait.patch --]
[-- Type: text/plain, Size: 7874 bytes --]

Introduce super_block.s_more_io_wait and writeback_control.more_io_wait.
They are for inodes that for some reason cannot be synced immediately.

These inodes will be moved to s_more_io_wait and retried after a while(waiting
up to 0.1s).  The normal lots-of-dirty-pages inodes will be moved to more_io
and be synced after other inodes in s_io have been serviced(no sleep).

The new data flow is now simple and fair:
- to fill s_io:
		s_more_io +
		s_dirty(expired) +
		s_more_io_wait
				---> s_io
- to drain s_io:
		s_io -+--> clean inodes in inode_in_use/inode_unused
		      |
		      +--> s_more_io
		      |
		      +--> s_more_io_wait

- s_dirty is now a strict FIFO queue
- inode.dirtied_when now really means the first dirty time
- once exipired, the dirty inode will stay in s_*io* queues until made clean
- the dirty inodes in s_*io* queues will be revisted in order: no starvation

Cc: Michael Rubin <mrubin@google.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Fengguang Wu <wfg@mail.ustc.edu.cn>
---
 fs/fs-writeback.c         |   22 +++++++++++---
 fs/super.c                |    1 
 include/linux/fs.h        |    1 
 include/linux/writeback.h |    1 
 mm/page-writeback.c       |   56 ++++++++++++++++++++----------------
 5 files changed, 53 insertions(+), 28 deletions(-)

--- linux-2.6.24-rc6-mm1.orig/fs/fs-writeback.c
+++ linux-2.6.24-rc6-mm1/fs/fs-writeback.c
@@ -170,10 +170,18 @@ static void redirty_tail(struct inode *i
 static void requeue_io(struct inode *inode)
 {
 	list_move(&inode->i_list, &inode->i_sb->s_more_io);
 }
 
+/*
+ * The inode should be retried after _sleeping_ for a while.
+ */
+static void requeue_io_wait(struct inode *inode)
+{
+	list_move(&inode->i_list, &inode->i_sb->s_more_io_wait);
+}
+
 static void inode_sync_complete(struct inode *inode)
 {
 	/*
 	 * Prevent speculative execution through spin_unlock(&inode_lock);
 	 */
@@ -204,17 +212,19 @@ static void move_expired_inodes(struct l
 static void queue_io(struct super_block *sb,
 				unsigned long *older_than_this)
 {
 	list_splice_init(&sb->s_more_io, sb->s_io.prev);
 	move_expired_inodes(&sb->s_dirty, &sb->s_io, older_than_this);
+	list_splice_init(&sb->s_more_io_wait, sb->s_io.prev);
 }
 
 int sb_has_dirty_inodes(struct super_block *sb)
 {
-	return !list_empty(&sb->s_dirty) ||
-	       !list_empty(&sb->s_io) ||
-	       !list_empty(&sb->s_more_io);
+	return !list_empty(&sb->s_dirty)   ||
+	       !list_empty(&sb->s_io)      ||
+	       !list_empty(&sb->s_more_io) ||
+	       !list_empty(&sb->s_more_io_wait);
 }
 EXPORT_SYMBOL(sb_has_dirty_inodes);
 
 /*
  * Write a single inode's dirty pages and inode data out to disk.
@@ -470,15 +480,19 @@ int generic_sync_sb_inodes(struct super_
 		}
 		spin_unlock(&inode_lock);
 		iput(inode);
 		cond_resched();
 		spin_lock(&inode_lock);
-		if (wbc->nr_to_write <= 0)
+		if (wbc->nr_to_write <= 0) {
+			wbc->more_io = 1;
 			break;
+		}
 	}
 	if (!list_empty(&sb->s_more_io))
 		wbc->more_io = 1;
+	if (!list_empty(&sb->s_more_io_wait))
+		wbc->more_io_wait = 1;
 	spin_unlock(&inode_lock);
 	return ret;		/* Leave any unwritten inodes on s_io */
 }
 EXPORT_SYMBOL(generic_sync_sb_inodes);
 
--- linux-2.6.24-rc6-mm1.orig/mm/page-writeback.c
+++ linux-2.6.24-rc6-mm1/mm/page-writeback.c
@@ -541,21 +541,48 @@ void throttle_vm_writeout(gfp_t gfp_mask
 			break;
         }
 }
 
 /*
+ * Write back up to MAX_WRITEBACK_PAGES.
+ * Return true if there's no more work.
+ */
+static int writeback_some_pages(struct writeback_control *wbc, int nr)
+{
+	int all_done = 0;
+
+	wbc->more_io = 0;
+	wbc->more_io_wait = 0;
+	wbc->encountered_congestion = 0;
+	wbc->nr_to_write = nr;
+
+	writeback_inodes(wbc);
+
+	if (wbc->encountered_congestion)
+		congestion_wait(WRITE, HZ/10);
+
+	if (wbc->more_io)
+		;
+	else if (wbc->more_io_wait)
+		congestion_wait(WRITE, HZ/10);
+	else
+		all_done = 1;
+
+	return all_done;
+}
+
+/*
  * writeback at least _min_pages, and keep writing until the amount of dirty
  * memory is less than the background threshold, or until we're all clean.
  */
 static void background_writeout(unsigned long _min_pages)
 {
 	long min_pages = _min_pages;
 	struct writeback_control wbc = {
 		.bdi		= NULL,
 		.sync_mode	= WB_SYNC_NONE,
 		.older_than_this = NULL,
-		.nr_to_write	= 0,
 		.nonblocking	= 1,
 		.range_cyclic	= 1,
 	};
 
 	for ( ; ; ) {
@@ -565,23 +592,13 @@ static void background_writeout(unsigned
 		get_dirty_limits(&background_thresh, &dirty_thresh, NULL, NULL);
 		if (global_page_state(NR_FILE_DIRTY) +
 			global_page_state(NR_UNSTABLE_NFS) < background_thresh
 				&& min_pages <= 0)
 			break;
-		wbc.more_io = 0;
-		wbc.encountered_congestion = 0;
-		wbc.nr_to_write = MAX_WRITEBACK_PAGES;
-		wbc.pages_skipped = 0;
-		writeback_inodes(&wbc);
+		if (writeback_some_pages(&wbc, MAX_WRITEBACK_PAGES))
+			break;
 		min_pages -= MAX_WRITEBACK_PAGES - wbc.nr_to_write;
-		if (wbc.nr_to_write > 0 || wbc.pages_skipped > 0) {
-			/* Wrote less than expected */
-			if (wbc.encountered_congestion || wbc.more_io)
-				congestion_wait(WRITE, HZ/10);
-			else
-				break;
-		}
 	}
 }
 
 /*
  * Start writeback of `nr_pages' pages.  If `nr_pages' is zero, write back
@@ -625,11 +642,10 @@ static void wb_kupdate(unsigned long arg
 	long nr_to_write;
 	struct writeback_control wbc = {
 		.bdi		= NULL,
 		.sync_mode	= WB_SYNC_NONE,
 		.older_than_this = &oldest_jif,
-		.nr_to_write	= 0,
 		.nonblocking	= 1,
 		.for_kupdate	= 1,
 		.range_cyclic	= 1,
 	};
 
@@ -640,20 +656,12 @@ static void wb_kupdate(unsigned long arg
 	next_jif = start_jif + dirty_writeback_interval;
 	nr_to_write = global_page_state(NR_FILE_DIRTY) +
 			global_page_state(NR_UNSTABLE_NFS) +
 			(inodes_stat.nr_inodes - inodes_stat.nr_unused);
 	while (nr_to_write > 0) {
-		wbc.more_io = 0;
-		wbc.encountered_congestion = 0;
-		wbc.nr_to_write = MAX_WRITEBACK_PAGES;
-		writeback_inodes(&wbc);
-		if (wbc.nr_to_write > 0) {
-			if (wbc.encountered_congestion || wbc.more_io)
-				congestion_wait(WRITE, HZ/10);
-			else
-				break;	/* All the old data is written */
-		}
+		if (writeback_some_pages(&wbc, MAX_WRITEBACK_PAGES))
+			break;
 		nr_to_write -= MAX_WRITEBACK_PAGES - wbc.nr_to_write;
 	}
 	if (time_before(next_jif, jiffies + HZ))
 		next_jif = jiffies + HZ;
 	if (dirty_writeback_interval)
--- linux-2.6.24-rc6-mm1.orig/fs/super.c
+++ linux-2.6.24-rc6-mm1/fs/super.c
@@ -62,10 +62,11 @@ static struct super_block *alloc_super(s
 			goto out;
 		}
 		INIT_LIST_HEAD(&s->s_dirty);
 		INIT_LIST_HEAD(&s->s_io);
 		INIT_LIST_HEAD(&s->s_more_io);
+		INIT_LIST_HEAD(&s->s_more_io_wait);
 		INIT_LIST_HEAD(&s->s_files);
 		INIT_LIST_HEAD(&s->s_instances);
 		INIT_HLIST_HEAD(&s->s_anon);
 		INIT_LIST_HEAD(&s->s_inodes);
 		init_rwsem(&s->s_umount);
--- linux-2.6.24-rc6-mm1.orig/include/linux/fs.h
+++ linux-2.6.24-rc6-mm1/include/linux/fs.h
@@ -1009,10 +1009,11 @@ struct super_block {
 
 	struct list_head	s_inodes;	/* all inodes */
 	struct list_head	s_dirty;	/* dirty inodes */
 	struct list_head	s_io;		/* parked for writeback */
 	struct list_head	s_more_io;	/* parked for more writeback */
+	struct list_head	s_more_io_wait;	/* parked for sleep-then-retry */
 	struct hlist_head	s_anon;		/* anonymous dentries for (nfs) exporting */
 	struct list_head	s_files;
 
 	struct block_device	*s_bdev;
 	struct mtd_info		*s_mtd;
--- linux-2.6.24-rc6-mm1.orig/include/linux/writeback.h
+++ linux-2.6.24-rc6-mm1/include/linux/writeback.h
@@ -61,10 +61,11 @@ struct writeback_control {
 	unsigned for_kupdate:1;		/* A kupdate writeback */
 	unsigned for_reclaim:1;		/* Invoked from the page allocator */
 	unsigned for_writepages:1;	/* This is a writepages() call */
 	unsigned range_cyclic:1;	/* range_start is cyclic */
 	unsigned more_io:1;		/* more io to be dispatched */
+	unsigned more_io_wait:1;	/* more io to be dispatched after a while */
 };
 
 /*
  * fs/fs-writeback.c
  */	

-- 

[PATCH 02/11] writeback: requeue_io_wait() on blocked kupdate

[Fengguang Wu]

[-- Attachment #1: writeback-more_io_wait-a.patch --]
[-- Type: text/plain, Size: 1076 bytes --]

Convert to requeue_io_wait() for case:

	kupdate cannot write all pages because of uncontrollable bad conditions.

Cc: Michael Rubin <mrubin@google.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Fengguang Wu <wfg@mail.ustc.edu.cn>
---
 fs/fs-writeback.c |    7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

--- linux-2.6.24-rc6-mm1.orig/fs/fs-writeback.c
+++ linux-2.6.24-rc6-mm1/fs/fs-writeback.c
@@ -292,11 +292,16 @@ __sync_single_inode(struct inode *inode,
 				 * For the kupdate function we move the inode
 				 * to s_more_io so it will get more writeout as
 				 * soon as the queue becomes uncongested.
 				 */
 				inode->i_state |= I_DIRTY_PAGES;
-				requeue_io(inode);
+				if (wbc->nr_to_write > 0)
+					/* somehow blocked: wait and retry */
+					requeue_io_wait(inode);
+				else
+					/* slice used up: queue for next turn */
+					requeue_io(inode);
 			} else {
 				/*
 				 * Otherwise fully redirty the inode so that
 				 * other inodes on this superblock will get some
 				 * writeout.  Otherwise heavy writing to one

-- 

[PATCH 03/11] writeback: replace redirty_tail() on more-pages-to-sync

[Fengguang Wu]

[-- Attachment #1: writeback-simplify-more-io.patch --]
[-- Type: text/plain, Size: 2419 bytes --]

Replace redirty_tail() with requeue_io()/requeue_io_wait() when
there are more pages to sync. There won't be starvation.

Cc: Michael Rubin <mrubin@google.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Fengguang Wu <wfg@mail.ustc.edu.cn>
---
 fs/fs-writeback.c |   43 ++++++++-----------------------------------
 1 file changed, 8 insertions(+), 35 deletions(-)

--- linux-2.6.24-rc6-mm1.orig/fs/fs-writeback.c
+++ linux-2.6.24-rc6-mm1/fs/fs-writeback.c
@@ -273,46 +273,19 @@ __sync_single_inode(struct inode *inode,
 	if (!(inode->i_state & I_FREEING)) {
 		if (!(inode->i_state & I_DIRTY) &&
 		    mapping_tagged(mapping, PAGECACHE_TAG_DIRTY)) {
 			/*
 			 * We didn't write back all the pages.  nfs_writepages()
-			 * sometimes bales out without doing anything. Redirty
-			 * the inode; Move it from s_io onto s_more_io/s_dirty.
+			 * sometimes bales out without doing anything.
 			 */
-			/*
-			 * akpm: if the caller was the kupdate function we put
-			 * this inode at the head of s_dirty so it gets first
-			 * consideration.  Otherwise, move it to the tail, for
-			 * the reasons described there.  I'm not really sure
-			 * how much sense this makes.  Presumably I had a good
-			 * reasons for doing it this way, and I'd rather not
-			 * muck with it at present.
-			 */
-			if (wbc->for_kupdate) {
-				/*
-				 * For the kupdate function we move the inode
-				 * to s_more_io so it will get more writeout as
-				 * soon as the queue becomes uncongested.
-				 */
-				inode->i_state |= I_DIRTY_PAGES;
-				if (wbc->nr_to_write > 0)
-					/* somehow blocked: wait and retry */
-					requeue_io_wait(inode);
-				else
-					/* slice used up: queue for next turn */
-					requeue_io(inode);
-			} else {
-				/*
-				 * Otherwise fully redirty the inode so that
-				 * other inodes on this superblock will get some
-				 * writeout.  Otherwise heavy writing to one
-				 * file would indefinitely suspend writeout of
-				 * all the other files.
-				 */
-				inode->i_state |= I_DIRTY_PAGES;
-				redirty_tail(inode);
-			}
+			inode->i_state |= I_DIRTY_PAGES;
+			if (wbc->nr_to_write > 0)
+				/* somehow blocked: wait and retry */
+				requeue_io_wait(inode);
+			else
+				/* slice used up: queue for next turn */
+				requeue_io(inode);
 		} else if (inode->i_state & I_DIRTY) {
 			/*
 			 * Someone redirtied the inode while were writing back
 			 * the pages.
 			 */

-- 

[PATCH 04/11] writeback: requeue_io_wait() on locked inode

[Fengguang Wu]

[-- Attachment #1: writeback-more_io_wait-b.patch --]
[-- Type: text/plain, Size: 1050 bytes --]

Convert to requeue_io_wait() for case:

	inode is locked

Cc: Michael Rubin <mrubin@google.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Fengguang Wu <wfg@mail.ustc.edu.cn>
---
 fs/fs-writeback.c |    7 ++-----
 1 file changed, 2 insertions(+), 5 deletions(-)

--- linux-2.6.24-rc6-mm1.orig/fs/fs-writeback.c
+++ linux-2.6.24-rc6-mm1/fs/fs-writeback.c
@@ -322,16 +322,13 @@ __writeback_single_inode(struct inode *i
 		WARN_ON(inode->i_state & I_WILL_FREE);
 
 	if ((wbc->sync_mode != WB_SYNC_ALL) && (inode->i_state & I_SYNC)) {
 		/*
 		 * We're skipping this inode because it's locked, and we're not
-		 * doing writeback-for-data-integrity.  Move it to s_more_io so
-		 * that writeback can proceed with the other inodes on s_io.
-		 * We'll have another go at writing back this inode when we
-		 * completed a full scan of s_io.
+		 * doing writeback-for-data-integrity. Recheck it after a while.
 		 */
-		requeue_io(inode);
+		requeue_io_wait(inode);
 		return 0;
 	}
 
 	/*
 	 * It's a data-integrity sync.  We must wait.

-- 

[PATCH 05/11] writeback: requeue_io_wait() on congested blockdev

[Fengguang Wu]

[-- Attachment #1: writeback-more_io_wait-c.patch --]
[-- Type: text/plain, Size: 768 bytes --]

Convert to requeue_io_wait() for case:

	congested blockdev

Cc: Michael Rubin <mrubin@google.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Fengguang Wu <wfg@mail.ustc.edu.cn>
---
 fs/fs-writeback.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

--- linux-2.6.24-rc6-mm1.orig/fs/fs-writeback.c
+++ linux-2.6.24-rc6-mm1/fs/fs-writeback.c
@@ -413,11 +413,11 @@ int generic_sync_sb_inodes(struct super_
 
 		if (wbc->nonblocking && bdi_write_congested(bdi)) {
 			wbc->encountered_congestion = 1;
 			if (!sb_is_blkdev_sb(sb))
 				break;		/* Skip a congested fs */
-			requeue_io(inode);
+			requeue_io_wait(inode);
 			continue;		/* Skip a congested blockdev */
 		}
 
 		if (wbc->bdi && bdi != wbc->bdi) {
 			if (!sb_is_blkdev_sb(sb))

-- 

[PATCH 06/11] writeback: requeue_io_wait() on locked buffers

[Fengguang Wu]

[-- Attachment #1: writeback-more_io_wait-d.patch --]
[-- Type: text/plain, Size: 755 bytes --]

Convert to requeue_io_wait() for case:

	pages skipped due to locked buffers

Cc: Michael Rubin <mrubin@google.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Fengguang Wu <wfg@mail.ustc.edu.cn>
---
 fs/fs-writeback.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

--- linux-2.6.24-rc6-mm1.orig/fs/fs-writeback.c
+++ linux-2.6.24-rc6-mm1/fs/fs-writeback.c
@@ -449,11 +449,11 @@ int generic_sync_sb_inodes(struct super_
 		if (wbc->pages_skipped != pages_skipped) {
 			/*
 			 * writeback is not making progress due to locked
 			 * buffers.  Skip this inode for now.
 			 */
-			redirty_tail(inode);
+			requeue_io_wait(inode);
 		}
 		spin_unlock(&inode_lock);
 		iput(inode);
 		cond_resched();
 		spin_lock(&inode_lock);

-- 

[PATCH 07/11] writeback: requeue_io() on redirtied inode

[Fengguang Wu]

[-- Attachment #1: writeback-remove-redirty-b.patch --]
[-- Type: text/plain, Size: 897 bytes --]

Redirtied inodes could be seen in really fast writes.
They should really be synced as soon as possible.

redirty_tail() could delay the inode for up to 30s.
Kill the delay by using requeue_io() instead.

Cc: Michael Rubin <mrubin@google.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Fengguang Wu <wfg@mail.ustc.edu.cn>
---
 fs/fs-writeback.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

--- linux-2.6.24-rc6-mm1.orig/fs/fs-writeback.c
+++ linux-2.6.24-rc6-mm1/fs/fs-writeback.c
@@ -287,11 +287,11 @@ __sync_single_inode(struct inode *inode,
 		} else if (inode->i_state & I_DIRTY) {
 			/*
 			 * Someone redirtied the inode while were writing back
 			 * the pages.
 			 */
-			redirty_tail(inode);
+			requeue_io(inode);
 		} else if (atomic_read(&inode->i_count)) {
 			/*
 			 * The inode is clean, inuse
 			 */
 			list_move(&inode->i_list, &inode_in_use);

-- 

[PATCH 08/11] writeback: introduce queue_dirty()

[Fengguang Wu]

[-- Attachment #1: writeback-queue_dirty.patch --]
[-- Type: text/plain, Size: 1876 bytes --]

Introduce queue_dirty() to enqueue a newly dirtied inode.
It helps remove duplicate code.

Cc: Michael Rubin <mrubin@google.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Fengguang Wu <wfg@mail.ustc.edu.cn>
---
 fs/fs-writeback.c |   21 +++++++++++++--------
 1 file changed, 13 insertions(+), 8 deletions(-)

--- linux-2.6.24-rc6-mm1.orig/fs/fs-writeback.c
+++ linux-2.6.24-rc6-mm1/fs/fs-writeback.c
@@ -23,10 +23,19 @@
 #include <linux/blkdev.h>
 #include <linux/backing-dev.h>
 #include <linux/buffer_head.h>
 #include "internal.h"
 
+/*
+ * Enqueue a newly dirtied inode.
+ */
+static void queue_dirty(struct inode *inode)
+{
+	inode->dirtied_when = jiffies;
+	list_move(&inode->i_list, &inode->i_sb->s_dirty);
+}
+
 /**
  *	__mark_inode_dirty -	internal function
  *	@inode: inode to mark
  *	@flags: what kind of dirty (i.e. I_DIRTY_SYNC)
  *	Mark an inode as dirty. Callers should use mark_inode_dirty or
@@ -120,14 +129,12 @@ void __mark_inode_dirty(struct inode *in
 
 		/*
 		 * If the inode was already on s_dirty/s_io/s_more_io, don't
 		 * reposition it (that would break s_dirty time-ordering).
 		 */
-		if (!was_dirty) {
-			inode->dirtied_when = jiffies;
-			list_move(&inode->i_list, &sb->s_dirty);
-		}
+		if (!was_dirty)
+			queue_dirty(inode);
 	}
 out:
 	spin_unlock(&inode_lock);
 }
 
@@ -438,14 +445,12 @@ int generic_sync_sb_inodes(struct super_
 		__iget(inode);
 		pages_skipped = wbc->pages_skipped;
 		err = __writeback_single_inode(inode, wbc);
 		if (!ret)
 			ret = err;
-		if (wbc->sync_mode == WB_SYNC_HOLD) {
-			inode->dirtied_when = jiffies;
-			list_move(&inode->i_list, &sb->s_dirty);
-		}
+		if (wbc->sync_mode == WB_SYNC_HOLD)
+			queue_dirty(inode);
 		if (current_is_pdflush())
 			writeback_release(bdi);
 		if (wbc->pages_skipped != pages_skipped) {
 			/*
 			 * writeback is not making progress due to locked

-- 

[PATCH 09/11] writeback: replace redirty_tail() on memory-backed bdi

[Fengguang Wu]

[-- Attachment #1: writeback-remove-redirty-a.patch --]
[-- Type: text/plain, Size: 769 bytes --]

Replace redirty_tail() with queue_dirty() on memory backed bdi.

Cc: Michael Rubin <mrubin@google.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Fengguang Wu <wfg@mail.ustc.edu.cn>
---
 fs/fs-writeback.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

--- linux-2.6.24-rc6-mm1.orig/fs/fs-writeback.c
+++ linux-2.6.24-rc6-mm1/fs/fs-writeback.c
@@ -400,11 +400,11 @@ int generic_sync_sb_inodes(struct super_
 		struct backing_dev_info *bdi = mapping->backing_dev_info;
 		long pages_skipped;
 		int err;
 
 		if (!bdi_cap_writeback_dirty(bdi)) {
-			redirty_tail(inode);
+			queue_dirty(inode);
 			if (sb_is_blkdev_sb(sb)) {
 				/*
 				 * Dirty memory-backed blockdev: the ramdisk
 				 * driver does this.  Skip just this inode
 				 */

-- 

[PATCH 10/11] writeback: remove redirty_tail()

[Fengguang Wu]

[-- Attachment #1: writeback-remove-redirty_tail.patch --]
[-- Type: text/plain, Size: 1490 bytes --]

Remove redirty_tail(). It's no longer used.

Cc: Michael Rubin <mrubin@google.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Fengguang Wu <wfg@mail.ustc.edu.cn>
---
 fs/fs-writeback.c |   24 ------------------------
 1 file changed, 24 deletions(-)

--- linux-2.6.24-rc6-mm1.orig/fs/fs-writeback.c
+++ linux-2.6.24-rc6-mm1/fs/fs-writeback.c
@@ -146,34 +146,10 @@ static int write_inode(struct inode *ino
 		return inode->i_sb->s_op->write_inode(inode, sync);
 	return 0;
 }
 
 /*
- * Redirty an inode: set its when-it-was dirtied timestamp and move it to the
- * furthest end of its superblock's dirty-inode list.
- *
- * Before stamping the inode's ->dirtied_when, we check to see whether it is
- * already the most-recently-dirtied inode on the s_dirty list.  If that is
- * the case then the inode must have been redirtied while it was being written
- * out and we don't reset its dirtied_when.
- */
-static void redirty_tail(struct inode *inode)
-{
-	struct super_block *sb = inode->i_sb;
-
-	if (!list_empty(&sb->s_dirty)) {
-		struct inode *tail_inode;
-
-		tail_inode = list_entry(sb->s_dirty.next, struct inode, i_list);
-		if (!time_after_eq(inode->dirtied_when,
-				tail_inode->dirtied_when))
-			inode->dirtied_when = jiffies;
-	}
-	list_move(&inode->i_list, &sb->s_dirty);
-}
-
-/*
  * requeue inode for re-scanning after sb->s_io list is exhausted.
  */
 static void requeue_io(struct inode *inode)
 {
 	list_move(&inode->i_list, &inode->i_sb->s_more_io);

-- 

[PATCH 11/11] writeback: __sync_single_inode() code cleanup

[Fengguang Wu]

[-- Attachment #1: writeback-simplify-x.patch --]
[-- Type: text/plain, Size: 1654 bytes --]

Make the if-else straight in __sync_single_inode().
No behavior change.

Cc: Michael Rubin <mrubin@google.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Fengguang Wu <wfg@mail.ustc.edu.cn>
---
 fs/fs-writeback.c |   15 +++++++--------
 1 file changed, 7 insertions(+), 8 deletions(-)

--- linux-2.6.24-rc6-mm1.orig/fs/fs-writeback.c
+++ linux-2.6.24-rc6-mm1/fs/fs-writeback.c
@@ -248,37 +248,36 @@ __sync_single_inode(struct inode *inode,
 	if (wait) {
 		int err = filemap_fdatawait(mapping);
 		if (ret == 0)
 			ret = err;
 	}
 
 	spin_lock(&inode_lock);
 	inode->i_state &= ~I_SYNC;
 	if (!(inode->i_state & I_FREEING)) {
-		if (!(inode->i_state & I_DIRTY) &&
-		    mapping_tagged(mapping, PAGECACHE_TAG_DIRTY)) {
+		if (inode->i_state & I_DIRTY) {
+			/*
+			 * Someone redirtied the inode while were writing back
+			 * the pages.
+			 */
+			requeue_io(inode);
+		} else if (mapping_tagged(mapping, PAGECACHE_TAG_DIRTY)) {
 			/*
 			 * We didn't write back all the pages.  nfs_writepages()
 			 * sometimes bales out without doing anything.
 			 */
 			inode->i_state |= I_DIRTY_PAGES;
 			if (wbc->nr_to_write > 0)
 				/* somehow blocked: wait and retry */
 				requeue_io_wait(inode);
 			else
 				/* slice used up: queue for next turn */
 				requeue_io(inode);
-		} else if (inode->i_state & I_DIRTY) {
-			/*
-			 * Someone redirtied the inode while were writing back
-			 * the pages.
-			 */
-			requeue_io(inode);
 		} else if (atomic_read(&inode->i_count)) {
 			/*
 			 * The inode is clean, inuse
 			 */
 			list_move(&inode->i_list, &inode_in_use);
 		} else {
 			/*
 			 * The inode is clean, unused
 			 */

-- 

Re: [PATCH 00/11] writeback bug fixes and simplifications

[Andrew Morton]

On Thu, 27 Dec 2007 23:08:40 +0100 Sascha Warner <prx@cinatas.ath.cx> wrote:

> Hi,
> 
> I applied your patches to 2.6.24-rc6-mm1, but now I am faced with one
> pdflush often using 100% CPU for a long time. There seem to be some rare
> pauses from its 100% usage, however.
> 
> On ~23 minutes uptime i have ~19 minutes pdflush runtime.
> 
> This is on E6600, x86_64, 2 Gig RAM, SATA HDD, running on gentoo ~x64_64
> 
> Let me know if you need more info.
> 

(some) cc's restored.  Please, always do reply-to-all.

Re: [PATCH 00/11] writeback bug fixes and simplifications

[WU Fengguang]

On Fri, Dec 28, 2007 at 03:01:11PM -0800, Andrew Morton wrote:
> On Thu, 27 Dec 2007 23:08:40 +0100 Sascha Warner <prx@cinatas.ath.cx> wrote:
> 
> > Hi,
> > 
> > I applied your patches to 2.6.24-rc6-mm1, but now I am faced with one
> > pdflush often using 100% CPU for a long time. There seem to be some rare
> > pauses from its 100% usage, however.
> > 
> > On ~23 minutes uptime i have ~19 minutes pdflush runtime.
> > 
> > This is on E6600, x86_64, 2 Gig RAM, SATA HDD, running on gentoo ~x64_64
> > 
> > Let me know if you need more info.

Sascha, Thank you for the testing.

Replace the first patch with this one should help:

---
Subject: writeback: introduce s_more_io_wait for inodes to be re-synced after a while

Introduce super_block.s_more_io_wait and writeback_control.more_io_wait.
They are for inodes that for some reason cannot be synced immediately.

These inodes will be moved to s_more_io_wait and retried after a while(waiting
up to 0.1s).  The normal lots-of-dirty-pages inodes will be moved to more_io
and be synced after other inodes in s_io have been serviced(no sleep).

The data flow is made more simple:

s_dirty --> s_io --> s_more_io/s_more_io_wait --+
             ^                                  |
	     |                                  |
	     +----------------------------------+

- to fill s_io:
		s_more_io +
		s_dirty(expired) +
		s_more_io_wait
				---> s_io
- to drain s_io:
		s_io -+--> clean inodes goto inode_in_use/inode_unused
		      |
		      +--> s_more_io
		      |
		      +--> s_more_io_wait

Obviously there's no ordering or starvation problems in the queues:
- s_dirty is now a strict FIFO queue
- inode.dirtied_when now really means the first dirty time
- once exipired, the dirty inode will stay in s_*io* queues until made clean
- the dirty inodes in s_*io* queues will be revisted in order: no starvation

Cc: Michael Rubin <mrubin@google.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Fengguang Wu <wfg@mail.ustc.edu.cn>
---
 fs/fs-writeback.c         |   26 ++++++++++++----
 fs/super.c                |    1 
 include/linux/fs.h        |    1 
 include/linux/writeback.h |    1 
 mm/page-writeback.c       |   56 ++++++++++++++++++++----------------
 5 files changed, 55 insertions(+), 30 deletions(-)

--- linux-2.6.24-rc6-mm1.orig/fs/fs-writeback.c
+++ linux-2.6.24-rc6-mm1/fs/fs-writeback.c
@@ -172,6 +172,14 @@ static void requeue_io(struct inode *ino
 	list_move(&inode->i_list, &inode->i_sb->s_more_io);
 }
 
+/*
+ * The inode should be retried after _sleeping_ for a while.
+ */
+static void requeue_io_wait(struct inode *inode)
+{
+	list_move(&inode->i_list, &inode->i_sb->s_more_io_wait);
+}
+
 static void inode_sync_complete(struct inode *inode)
 {
 	/*
@@ -206,13 +214,15 @@ static void queue_io(struct super_block 
 {
 	list_splice_init(&sb->s_more_io, sb->s_io.prev);
 	move_expired_inodes(&sb->s_dirty, &sb->s_io, older_than_this);
+	list_splice_init(&sb->s_more_io_wait, sb->s_io.prev);
 }
 
 int sb_has_dirty_inodes(struct super_block *sb)
 {
-	return !list_empty(&sb->s_dirty) ||
-	       !list_empty(&sb->s_io) ||
-	       !list_empty(&sb->s_more_io);
+	return !list_empty(&sb->s_dirty)   ||
+	       !list_empty(&sb->s_io)      ||
+	       !list_empty(&sb->s_more_io) ||
+	       !list_empty(&sb->s_more_io_wait);
 }
 EXPORT_SYMBOL(sb_has_dirty_inodes);
 
@@ -472,11 +482,15 @@ int generic_sync_sb_inodes(struct super_
 		iput(inode);
 		cond_resched();
 		spin_lock(&inode_lock);
-		if (wbc->nr_to_write <= 0)
+		if (wbc->nr_to_write <= 0) {
+			wbc->more_io = 1;
 			break;
+		}
+		if (!list_empty(&sb->s_more_io))
+			wbc->more_io = 1;
+		if (!list_empty(&sb->s_more_io_wait))
+			wbc->more_io_wait = 1;
 	}
-	if (!list_empty(&sb->s_more_io))
-		wbc->more_io = 1;
 	spin_unlock(&inode_lock);
 	return ret;		/* Leave any unwritten inodes on s_io */
 }
--- linux-2.6.24-rc6-mm1.orig/mm/page-writeback.c
+++ linux-2.6.24-rc6-mm1/mm/page-writeback.c
@@ -543,6 +543,34 @@ void throttle_vm_writeout(gfp_t gfp_mask
 }
 
 /*
+ * Write back up to MAX_WRITEBACK_PAGES.
+ * Return true if there's no more work.
+ */
+static int writeback_some_pages(struct writeback_control *wbc, int nr)
+{
+	int all_done = 0;
+
+	wbc->more_io = 0;
+	wbc->more_io_wait = 0;
+	wbc->encountered_congestion = 0;
+	wbc->nr_to_write = nr;
+
+	writeback_inodes(wbc);
+
+	if (wbc->encountered_congestion)
+		congestion_wait(WRITE, HZ/10);
+
+	if (wbc->more_io)
+		;
+	else if (wbc->more_io_wait)
+		congestion_wait(WRITE, HZ/10);
+	else
+		all_done = 1;
+
+	return all_done;
+}
+
+/*
  * writeback at least _min_pages, and keep writing until the amount of dirty
  * memory is less than the background threshold, or until we're all clean.
  */
@@ -553,7 +581,6 @@ static void background_writeout(unsigned
 		.bdi		= NULL,
 		.sync_mode	= WB_SYNC_NONE,
 		.older_than_this = NULL,
-		.nr_to_write	= 0,
 		.nonblocking	= 1,
 		.range_cyclic	= 1,
 	};
@@ -567,19 +594,9 @@ static void background_writeout(unsigned
 			global_page_state(NR_UNSTABLE_NFS) < background_thresh
 				&& min_pages <= 0)
 			break;
-		wbc.more_io = 0;
-		wbc.encountered_congestion = 0;
-		wbc.nr_to_write = MAX_WRITEBACK_PAGES;
-		wbc.pages_skipped = 0;
-		writeback_inodes(&wbc);
+		if (writeback_some_pages(&wbc, MAX_WRITEBACK_PAGES))
+			break;
 		min_pages -= MAX_WRITEBACK_PAGES - wbc.nr_to_write;
-		if (wbc.nr_to_write > 0 || wbc.pages_skipped > 0) {
-			/* Wrote less than expected */
-			if (wbc.encountered_congestion || wbc.more_io)
-				congestion_wait(WRITE, HZ/10);
-			else
-				break;
-		}
 	}
 }
 
@@ -627,7 +644,6 @@ static void wb_kupdate(unsigned long arg
 		.bdi		= NULL,
 		.sync_mode	= WB_SYNC_NONE,
 		.older_than_this = &oldest_jif,
-		.nr_to_write	= 0,
 		.nonblocking	= 1,
 		.for_kupdate	= 1,
 		.range_cyclic	= 1,
@@ -642,16 +658,8 @@ static void wb_kupdate(unsigned long arg
 			global_page_state(NR_UNSTABLE_NFS) +
 			(inodes_stat.nr_inodes - inodes_stat.nr_unused);
 	while (nr_to_write > 0) {
-		wbc.more_io = 0;
-		wbc.encountered_congestion = 0;
-		wbc.nr_to_write = MAX_WRITEBACK_PAGES;
-		writeback_inodes(&wbc);
-		if (wbc.nr_to_write > 0) {
-			if (wbc.encountered_congestion || wbc.more_io)
-				congestion_wait(WRITE, HZ/10);
-			else
-				break;	/* All the old data is written */
-		}
+		if (writeback_some_pages(&wbc, MAX_WRITEBACK_PAGES))
+			break;
 		nr_to_write -= MAX_WRITEBACK_PAGES - wbc.nr_to_write;
 	}
 	if (time_before(next_jif, jiffies + HZ))
--- linux-2.6.24-rc6-mm1.orig/fs/super.c
+++ linux-2.6.24-rc6-mm1/fs/super.c
@@ -64,6 +64,7 @@ static struct super_block *alloc_super(s
 		INIT_LIST_HEAD(&s->s_dirty);
 		INIT_LIST_HEAD(&s->s_io);
 		INIT_LIST_HEAD(&s->s_more_io);
+		INIT_LIST_HEAD(&s->s_more_io_wait);
 		INIT_LIST_HEAD(&s->s_files);
 		INIT_LIST_HEAD(&s->s_instances);
 		INIT_HLIST_HEAD(&s->s_anon);
--- linux-2.6.24-rc6-mm1.orig/include/linux/fs.h
+++ linux-2.6.24-rc6-mm1/include/linux/fs.h
@@ -1011,6 +1011,7 @@ struct super_block {
 	struct list_head	s_dirty;	/* dirty inodes */
 	struct list_head	s_io;		/* parked for writeback */
 	struct list_head	s_more_io;	/* parked for more writeback */
+	struct list_head	s_more_io_wait;	/* parked for sleep-then-retry */
 	struct hlist_head	s_anon;		/* anonymous dentries for (nfs) exporting */
 	struct list_head	s_files;
 
--- linux-2.6.24-rc6-mm1.orig/include/linux/writeback.h
+++ linux-2.6.24-rc6-mm1/include/linux/writeback.h
@@ -63,6 +63,7 @@ struct writeback_control {
 	unsigned for_writepages:1;	/* This is a writepages() call */
 	unsigned range_cyclic:1;	/* range_start is cyclic */
 	unsigned more_io:1;		/* more io to be dispatched */
+	unsigned more_io_wait:1;	/* more io to be dispatched after a while */
 };
 
 /*

Re: [PATCH 00/11] writeback bug fixes and simplifications

[Sam Ravnborg]

> 
> P.S.: Andrew, sorry my subject pretended I was subscribed to lkml, I am not.
> I hoped for the script at lkml to pull the threads together because of
> the subject.

Find the relevant mail at lkml.org and have it forwarded to you.
Then you can do a proper reply-to-all without breaking the thread
and keeping relevant cc's.

	Sam

Re: [PATCH 00/11] writeback bug fixes and simplifications

[WU Fengguang]

On Fri, Dec 28, 2007 at 05:27:38AM +0100, Sascha Warner wrote:
> Andrew Morton wrote:
> > On Thu, 27 Dec 2007 23:08:40 +0100 Sascha Warner <prx@cinatas.ath.cx> wrote:
> >
> >   
> >> Hi,
> >>
> >> I applied your patches to 2.6.24-rc6-mm1, but now I am faced with one
> >> pdflush often using 100% CPU for a long time. There seem to be some rare
> >> pauses from its 100% usage, however.
> >>
> >> On ~23 minutes uptime i have ~19 minutes pdflush runtime.
> >>
> >> This is on E6600, x86_64, 2 Gig RAM, SATA HDD, running on gentoo ~x64_64
> >>
> >> Let me know if you need more info.
> >>
> >>     
> >
> > (some) cc's restored.  Please, always do reply-to-all.
> >   
> Hi Wu,
> 
> Your latest patch does fix the seen pdflush 100% issue :)

Thank you.

Andrew: I'll be off line for some days.
The patchset will be updated some time later.

Happy new year~

Fengguang

Re: [PATCH 00/11] writeback bug fixes and simplifications

[Hans-Peter Jansen]

Am Freitag, 28. Dezember 2007 schrieb Sascha Warner:
> Andrew Morton wrote:
> > On Thu, 27 Dec 2007 23:08:40 +0100 Sascha Warner <prx@cinatas.ath.cx> 
wrote:
> >> Hi,
> >>
> >> I applied your patches to 2.6.24-rc6-mm1, but now I am faced with one
> >> pdflush often using 100% CPU for a long time. There seem to be some
> >> rare pauses from its 100% usage, however.
> >>
> >> On ~23 minutes uptime i have ~19 minutes pdflush runtime.
> >>
> >> This is on E6600, x86_64, 2 Gig RAM, SATA HDD, running on gentoo
> >> ~x64_64
> >>
> >> Let me know if you need more info.
> >
> > (some) cc's restored.  Please, always do reply-to-all.
>
> Hi Wu,

Sascha, if you want to address Fengguang by his first name, note that 
chinese and bavarians (and some others I forgot now, too) typically use the 
order:
  
  lastname firstname 

when they spell their names. Another evidence is, that the name Wu is a 
pretty common chinese family name.

Fengguang, if it's the other way around, correct me please (and I'm going to 
wear a big brown paper bag for the rest of the day..). 

> Your latest patch does fix the seen pdflush 100% issue :)
>
> P.S.: Andrew, sorry my subject pretended I was subscribed to lkml, I am
> not. I hoped for the script at lkml to pull the threads together because
> of the subject.

Such a script _cannot_ exist for a mailing list. Sane mailing list software 
mangles messages in the least intrusive way and highly depend on users 
doing the right thing. In this case, that's what Sam suggested below!

A-happy-new-year-for-all-kernel-hacking-heros-ly-yrs,

Pete

Re: [PATCH 00/11] writeback bug fixes and simplifications

[WU Fengguang]

On Sat, Dec 29, 2007 at 03:56:59PM +0100, Hans-Peter Jansen wrote:
> Am Freitag, 28. Dezember 2007 schrieb Sascha Warner:
> > Andrew Morton wrote:
> > > On Thu, 27 Dec 2007 23:08:40 +0100 Sascha Warner <prx@cinatas.ath.cx> 
> wrote:
> > >> Hi,
> > >>
> > >> I applied your patches to 2.6.24-rc6-mm1, but now I am faced with one
> > >> pdflush often using 100% CPU for a long time. There seem to be some
> > >> rare pauses from its 100% usage, however.
> > >>
> > >> On ~23 minutes uptime i have ~19 minutes pdflush runtime.
> > >>
> > >> This is on E6600, x86_64, 2 Gig RAM, SATA HDD, running on gentoo
> > >> ~x64_64
> > >>
> > >> Let me know if you need more info.
> > >
> > > (some) cc's restored.  Please, always do reply-to-all.
> >
> > Hi Wu,
> 
> Sascha, if you want to address Fengguang by his first name, note that 
> chinese and bavarians (and some others I forgot now, too) typically use the 
> order:
>   
>   lastname firstname 
> 
> when they spell their names. Another evidence is, that the name Wu is a 
> pretty common chinese family name.
> 
> Fengguang, if it's the other way around, correct me please (and I'm going to 
> wear a big brown paper bag for the rest of the day..). 

You are right. We normally do "Fengguang" or "Mr. Wu" :-)
For LKML the first name is less ambiguous.

Thanks,
Fengguang

Re: [PATCH 00/11] writeback bug fixes and simplifications

[Martin Knoblauch]

----- Original Message ----
> From: WU Fengguang <wfg@mail.ustc.edu.cn>
> To: Hans-Peter Jansen <hpj@urpla.net>
> Cc: Sascha Warner <prx@cinatas.ath.cx>; Andrew Morton <akpm@linux-foundation.org>; linux-kernel@vger.kernel.org; Peter Zijlstra <a.p.zijlstra@chello.nl>
> Sent: Wednesday, January 9, 2008 4:33:32 AM
> Subject: Re: [PATCH 00/11] writeback bug fixes and simplifications
> 
> On Sat, Dec 29, 2007 at 03:56:59PM +0100, Hans-Peter Jansen wrote:
> > Am Freitag, 28. Dezember 2007 schrieb Sascha Warner:
> > > Andrew Morton wrote:
> > > > On Thu, 27 Dec 2007 23:08:40 +0100 Sascha
> Warner
> 
  
> > wrote:
> > > >> Hi,
> > > >>
> > > >> I applied your patches to 2.6.24-rc6-mm1, but now I am
> faced
> 
 with one
> > > >> pdflush often using 100% CPU for a long time. There seem to
> be
> 
 some
> > > >> rare pauses from its 100% usage, however.
> > > >>
> > > >> On ~23 minutes uptime i have ~19 minutes pdflush runtime.
> > > >>
> > > >> This is on E6600, x86_64, 2 Gig RAM, SATA HDD, running on gentoo
> > > >> ~x64_64
> > > >>
> > > >> Let me know if you need more info.
> > > >
> > > > (some) cc's restored.  Please, always do reply-to-all.
> > >
> > > Hi Wu,
> > 
> > Sascha, if you want to address Fengguang by his first name, note
> that
> 
 
> > chinese and bavarians (and some others I forgot now, too)
> typically
> 
 use the 
> > order:
> >   
> >   lastname firstname 
> > 
> > when they spell their names. Another evidence is, that the name Wu
> is
> 
 a 
> > pretty common chinese family name.
> > 
> > Fengguang, if it's the other way around, correct me please (and
> I'm
> 
 going to 
> > wear a big brown paper bag for the rest of the day..). 
> 
> You are right. We normally do "Fengguang" or "Mr. Wu" :-)
> For LKML the first name is less ambiguous.
> 
> Thanks,
> Fengguang
> 

 Just cannot resist. Hans-Peter mentions Bavarian using Lastname-Givenname as well. This is only true in a folklore context (or when you are very deep in the countryside). Officially the bavarians use the usual German Given/Lastname. Although they will never admit to be Germans, of course :-)


Cheers
Martin