2.5.56-mm1

2.5.56-mm1

[Andrew Morton]

http://www.zip.com.au/~akpm/linux/patches/2.5/2.5.56/2.5.56-mm1/

Nothing much new here except for a fix for the ext3-related memory leak which
Con reported recently.


The main items which remain unmerged from the -mm patch series are now:

- red/black-tree based insertion and sorting for the I/O scheduler.

  Jens will be submitting this next week.  It's completely stable, and the
  patch includes the addition of the I/O scheduler tunables in
  /sys/block/hda/iosched/, which is fairly important.

- Code to automatically unplug request queues on the basis of their
  occupancy and a timeout.

  Jens will be reviewing this soon.

- dcache-RCU.

  This was recently updated to fix a rename race.  It's quite stable.  I'm
  not sure where we stand wrt merging it now.  Al seems to have disappeared.

- Ingo Oeser's user page walking rework.  This appears to be stable,
  although I'm not sure what testing it has had apart from a lot of direct-io
  testing.

- Quite a lot of misc stuff which I need to go through and either send or
  toss.


Changes since 2.5.55-mm1:


+linus.patch

 Latest from Linus

-inlines-net.patch
-deadline-fixups.patch
-i_shared_sem.patch
-cond_resched_lock-rework.patch
-untypedef-mmu_gather.patch
-touched_by_munmap-go-forwards.patch
-low-latency-page-unmapping.patch
-misc.patch
-smp-preempt-latency-fix.patch
-set_page_dirty_lock.patch
-inline-constant-small-copy_user.patch

 Merged

+deadline-fixes.patch

 Some deadline scheduler tweaks and fixes

+deadline-sysfs-fix.patch

 Fix up the deadline scheduler patches to track recent sysfs changes

+ext3-leak-fix.patch

 Fix the memory leak whcih Con reported

+hugetlbfs-read-write.patch

 Don't permit reading or writing of hugetlbfs files.



All 44 patches:

linus.patch
  cset-1.897-to-1.929.txt.gz

kgdb.patch

rcf.patch
  run-child-first after fork

devfs-fix.patch

cputimes_stat.patch
  Retore per-cpu time accounting, with a config option

rbtree-iosched.patch
  rbtree-based IO scheduler

deadline-fixes.patch
  deadsched cleanups/fixups

deadline-sysfs-fix.patch

ext3-ino_t-cleanup.patch
  Subject: [PATCH] 2.5 ext3 ino_t removal

smaller-head-arrays.patch

setuid-exec-no-lock_kernel.patch
  remove lock_kernel() from exec of setuid apps

ptrace-flush.patch
  Subject: [PATCH] ptrace on 2.5.44

buffer-debug.patch
  buffer.c debugging

warn-null-wakeup.patch

pentium-II.patch
  Pentium-II support bits

reiserfs-readpages.patch
  reiserfs v3 readpages support

rcu-stats.patch
  RCU statistics reporting

auto-unplug.patch
  self-unplugging request queues

less-unplugging.patch
  Remove most of the blk_run_queues() calls

ext3-fsync-speedup.patch
  Clean up ext3_sync_file()

lockless-current_kernel_time.patch
  Lockless current_kernel_timer()

scheduler-tunables.patch
  scheduler tunables

htlb-2.patch
  hugetlb: fix MAP_FIXED handling

ext3-leak-fix.patch
  fix ext3 memory leak

hugetlbfs-read-write.patch
  hugetlbfs: don't implement read/write file_ops

oprofile-p4.patch

op4-fix.patch

wli-02_do_sak.patch
  (undescribed patch)

wli-03_proc_super.patch
  (undescribed patch)

wli-06_uml_get_task.patch
  (undescribed patch)

wli-07_numaq_mem_map.patch
  (undescribed patch)

wli-08_numaq_pgdat.patch
  (undescribed patch)

wli-09_has_stopped_jobs.patch
  (undescribed patch)

wli-10_inode_wait.patch
  (undescribed patch)

wli-11_pgd_ctor.patch
  (undescribed patch)

wli-11_pgd_ctor-update.patch
  pgd_ctor update

wli-13_rmap_nrpte.patch
  (undescribed patch)

dcache_rcu-2.patch
  dcache_rcu-2-2.5.51.patch

dcache_rcu-3.patch
  dcache_rcu-3-2.5.51.patch

page-walk-api.patch

page-walk-api-2.5.53-mm2-update.patch
  pagewalk API update

page-walk-scsi.patch

page-walk-scsi-2.5.53-mm2.patch
  pagewalk scsi update

smalldevfs.patch
  smalldevfs

2.5.56-mm1

[Andrew Morton]

http://www.zip.com.au/~akpm/linux/patches/2.5/2.5.56/2.5.56-mm1/

Nothing much new here except for a fix for the ext3-related memory leak which
Con reported recently.


The main items which remain unmerged from the -mm patch series are now:

- red/black-tree based insertion and sorting for the I/O scheduler.

  Jens will be submitting this next week.  It's completely stable, and the
  patch includes the addition of the I/O scheduler tunables in
  /sys/block/hda/iosched/, which is fairly important.

- Code to automatically unplug request queues on the basis of their
  occupancy and a timeout.

  Jens will be reviewing this soon.

- dcache-RCU.

  This was recently updated to fix a rename race.  It's quite stable.  I'm
  not sure where we stand wrt merging it now.  Al seems to have disappeared.

- Ingo Oeser's user page walking rework.  This appears to be stable,
  although I'm not sure what testing it has had apart from a lot of direct-io
  testing.

- Quite a lot of misc stuff which I need to go through and either send or
  toss.


Changes since 2.5.55-mm1:


+linus.patch

 Latest from Linus

-inlines-net.patch
-deadline-fixups.patch
-i_shared_sem.patch
-cond_resched_lock-rework.patch
-untypedef-mmu_gather.patch
-touched_by_munmap-go-forwards.patch
-low-latency-page-unmapping.patch
-misc.patch
-smp-preempt-latency-fix.patch
-set_page_dirty_lock.patch
-inline-constant-small-copy_user.patch

 Merged

+deadline-fixes.patch

 Some deadline scheduler tweaks and fixes

+deadline-sysfs-fix.patch

 Fix up the deadline scheduler patches to track recent sysfs changes

+ext3-leak-fix.patch

 Fix the memory leak whcih Con reported

+hugetlbfs-read-write.patch

 Don't permit reading or writing of hugetlbfs files.



All 44 patches:

linus.patch
  cset-1.897-to-1.929.txt.gz

kgdb.patch

rcf.patch
  run-child-first after fork

devfs-fix.patch

cputimes_stat.patch
  Retore per-cpu time accounting, with a config option

rbtree-iosched.patch
  rbtree-based IO scheduler

deadline-fixes.patch
  deadsched cleanups/fixups

deadline-sysfs-fix.patch

ext3-ino_t-cleanup.patch
  Subject: [PATCH] 2.5 ext3 ino_t removal

smaller-head-arrays.patch

setuid-exec-no-lock_kernel.patch
  remove lock_kernel() from exec of setuid apps

ptrace-flush.patch
  Subject: [PATCH] ptrace on 2.5.44

buffer-debug.patch
  buffer.c debugging

warn-null-wakeup.patch

pentium-II.patch
  Pentium-II support bits

reiserfs-readpages.patch
  reiserfs v3 readpages support

rcu-stats.patch
  RCU statistics reporting

auto-unplug.patch
  self-unplugging request queues

less-unplugging.patch
  Remove most of the blk_run_queues() calls

ext3-fsync-speedup.patch
  Clean up ext3_sync_file()

lockless-current_kernel_time.patch
  Lockless current_kernel_timer()

scheduler-tunables.patch
  scheduler tunables

htlb-2.patch
  hugetlb: fix MAP_FIXED handling

ext3-leak-fix.patch
  fix ext3 memory leak

hugetlbfs-read-write.patch
  hugetlbfs: don't implement read/write file_ops

oprofile-p4.patch

op4-fix.patch

wli-02_do_sak.patch
  (undescribed patch)

wli-03_proc_super.patch
  (undescribed patch)

wli-06_uml_get_task.patch
  (undescribed patch)

wli-07_numaq_mem_map.patch
  (undescribed patch)

wli-08_numaq_pgdat.patch
  (undescribed patch)

wli-09_has_stopped_jobs.patch
  (undescribed patch)

wli-10_inode_wait.patch
  (undescribed patch)

wli-11_pgd_ctor.patch
  (undescribed patch)

wli-11_pgd_ctor-update.patch
  pgd_ctor update

wli-13_rmap_nrpte.patch
  (undescribed patch)

dcache_rcu-2.patch
  dcache_rcu-2-2.5.51.patch

dcache_rcu-3.patch
  dcache_rcu-3-2.5.51.patch

page-walk-api.patch

page-walk-api-2.5.53-mm2-update.patch
  pagewalk API update

page-walk-scsi.patch

page-walk-scsi-2.5.53-mm2.patch
  pagewalk scsi update

smalldevfs.patch
  smalldevfs




--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: 2.5.56-mm1

[Jeff Garzik]

On Sat, Jan 11, 2003 at 02:43:08PM -0800, Andrew Morton wrote:
> - dcache-RCU.
> 
>   This was recently updated to fix a rename race.  It's quite stable.  I'm
>   not sure where we stand wrt merging it now.  Al seems to have disappeared.

I talked to him in person last week, and this was one of the topics of
discussion.  He seemed to think it was fundamentally unfixable.  He
proceed to explain why, and then explained the scheme he worked out to
improve things.  Unfortunately my memory cannot do justice to the
details.

Next time he explains it, I will write it down :)

Sorry for so lame a data point :)

	Jeff

Re: 2.5.56-mm1

[Jeff Garzik]

On Sat, Jan 11, 2003 at 02:43:08PM -0800, Andrew Morton wrote:
> - dcache-RCU.
> 
>   This was recently updated to fix a rename race.  It's quite stable.  I'm
>   not sure where we stand wrt merging it now.  Al seems to have disappeared.

I talked to him in person last week, and this was one of the topics of
discussion.  He seemed to think it was fundamentally unfixable.  He
proceed to explain why, and then explained the scheme he worked out to
improve things.  Unfortunately my memory cannot do justice to the
details.

Next time he explains it, I will write it down :)

Sorry for so lame a data point :)

	Jeff




--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: 2.5.56-mm1

[Dipankar Sarma]

On Sat, Jan 11, 2003 at 11:00:38PM +0000, Jeff Garzik wrote:
> On Sat, Jan 11, 2003 at 02:43:08PM -0800, Andrew Morton wrote:
> > - dcache-RCU.
> > 
> >   This was recently updated to fix a rename race.  It's quite stable.  I'm
> >   not sure where we stand wrt merging it now.  Al seems to have disappeared.
> 
> I talked to him in person last week, and this was one of the topics of
> discussion.  He seemed to think it was fundamentally unfixable.  He
> proceed to explain why, and then explained the scheme he worked out to
> improve things.  Unfortunately my memory cannot do justice to the
> details.

The rename race is fixed now. Yes, it was unfixable using *existing* RCU
techniques, but one has to invent new tricks when the old bag of
tricks is empty :)

Fundamentally what happens is that rename may be *two* updates - delete
from one hash chain and insert into another hash chain. In order for
lockfree traversal to work correctly, you must have a grace period after
each update. If we do a grace period between these two updates in a rename,
it slows down renames to unacceptable levels. So we had a problem there.

The solution lies in the dcache itself - it has a fast path (cached_lookup)
and a slow path (real_lookup). So all we had to do was to detect that a
rename had happened to the dentry while we looked it up lockfree. This
is done by a generation counter (d_move_count) in the dentry and is
protected by the per-dentry spinlock which we take during rename and
a successful cache lookup. 

Two things can happen due to the rename race - lookup incorrectly succeeds
or lookup incorrectly fails. The success case is easily handled by 
the lockfree lookup code that looks like this -

for the dentries in the hash chain {
	... More stuff....
	move_count = dentry->d_move_count;
	if (dentry name matches) {
		/* lookup succeeds */
		spin_lock(&dentry->d_lock);
		if (move_count != dentry->d_move_count) {
			/* 
			 * A rename happened while looking up lockfree and 
			 * we now cannot gurantee
			 * that the lookup is correct
			 */
			spin_unlock(&dentry->d_lock);
			return slow_lookup();
		}
		....
		....
	}
	... More stuff....
}

If the lookup fails due to rename race, then there will anyway be a
slow real_lookup which is serialized with rename.

Maneesh did a lot of testing using many ramfs and many millions of renames
with millions of lookups going on at the same time and slow path was hit only
100 times or so. For practical workloads, this should have absolutely no
performance impact.

Thanks
Dipankar

Re: 2.5.56-mm1

[Dipankar Sarma]

On Sat, Jan 11, 2003 at 11:00:38PM +0000, Jeff Garzik wrote:
> On Sat, Jan 11, 2003 at 02:43:08PM -0800, Andrew Morton wrote:
> > - dcache-RCU.
> > 
> >   This was recently updated to fix a rename race.  It's quite stable.  I'm
> >   not sure where we stand wrt merging it now.  Al seems to have disappeared.
> 
> I talked to him in person last week, and this was one of the topics of
> discussion.  He seemed to think it was fundamentally unfixable.  He
> proceed to explain why, and then explained the scheme he worked out to
> improve things.  Unfortunately my memory cannot do justice to the
> details.

The rename race is fixed now. Yes, it was unfixable using *existing* RCU
techniques, but one has to invent new tricks when the old bag of
tricks is empty :)

Fundamentally what happens is that rename may be *two* updates - delete
from one hash chain and insert into another hash chain. In order for
lockfree traversal to work correctly, you must have a grace period after
each update. If we do a grace period between these two updates in a rename,
it slows down renames to unacceptable levels. So we had a problem there.

The solution lies in the dcache itself - it has a fast path (cached_lookup)
and a slow path (real_lookup). So all we had to do was to detect that a
rename had happened to the dentry while we looked it up lockfree. This
is done by a generation counter (d_move_count) in the dentry and is
protected by the per-dentry spinlock which we take during rename and
a successful cache lookup. 

Two things can happen due to the rename race - lookup incorrectly succeeds
or lookup incorrectly fails. The success case is easily handled by 
the lockfree lookup code that looks like this -

for the dentries in the hash chain {
	... More stuff....
	move_count = dentry->d_move_count;
	if (dentry name matches) {
		/* lookup succeeds */
		spin_lock(&dentry->d_lock);
		if (move_count != dentry->d_move_count) {
			/* 
			 * A rename happened while looking up lockfree and 
			 * we now cannot gurantee
			 * that the lookup is correct
			 */
			spin_unlock(&dentry->d_lock);
			return slow_lookup();
		}
		....
		....
	}
	... More stuff....
}

If the lookup fails due to rename race, then there will anyway be a
slow real_lookup which is serialized with rename.

Maneesh did a lot of testing using many ramfs and many millions of renames
with millions of lookups going on at the same time and slow path was hit only
100 times or so. For practical workloads, this should have absolutely no
performance impact.

Thanks
Dipankar
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: 2.5.56-mm1

[William Lee Irwin III]

On Sat, Jan 11, 2003 at 02:43:08PM -0800, Andrew Morton wrote:
> wli-11_pgd_ctor.patch
>   (undescribed patch)
> wli-11_pgd_ctor-update.patch
>   pgd_ctor update

pgd_alloc() has officially vaporized off my profiles on my (UP) homebox.


Thanks,
Bill

Re: 2.5.56-mm1

[William Lee Irwin III]

On Sat, Jan 11, 2003 at 02:43:08PM -0800, Andrew Morton wrote:
> wli-11_pgd_ctor.patch
>   (undescribed patch)
> wli-11_pgd_ctor-update.patch
>   pgd_ctor update

pgd_alloc() has officially vaporized off my profiles on my (UP) homebox.


Thanks,
Bill
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/