2.5.68-mm2

2.5.68-mm2

[Andrew Morton]

http://www.zip.com.au/~akpm/linux/patches/2.5/2.5.68/2.5.68-mm2.gz

   Will appear sometime at:

ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.5/2.5.68/2.5.68-mm2/

. Zillions of new fixes.

. I got tired of the objrmap code going BUG under stress, so it is now in
  disgrace in the experimental/ directory.



Changes since 2.5.68-mm1:

+linus.patch

 Latest from Linus

-3c574-irq-fix.patch
-nec98-partitions-fix.patch
-dmfe-kfree_skb-fix.patch
-dentry_stat-accounting-fix.patch
-DCACHE_REFERENCED-fixes.patch
-tasklist_lock-dcache_lock-inversion-fix.patch
-setserial-fix.patch
-SAK-raw-mode-fix.patch
-pci-bus-ordering-fix.patch
-turn-on-NUMA-rebalancing.patch
-yellowfin-set_bit-fix.patch
-move-__set_page_dirty-buffers.patch
-buffers-cleanup.patch
-follow_hugetlb_page-fix.patch
-hugetlb-overflow-fix.patch
-mach64-build-fix.patch
-sync-all-quotas.patch
-aio-mmap-fix.patch
-shmdt-speedup.patch
-task_prio-fix.patch
-gfp_repeat.patch
-alloc_buffer_head-take-gfp.patch
-pte_alloc_one-use-gfp_repeat.patch
-pmd_alloc_one-use-gfp_repeat.patch
-overcommit-stop-swapoff.patch
-interruptible-swapoff.patch
-oomkill-swapoff.patch
-dac960-bounce-avoidance.patch
-shm_get_stat-handle-hugetlb-pages.patch
-dynamic-hd_struct-allocation.patch
-NOMMU-merge-fixes.patch
-vmap-extensions.patch
-dont-shrink-slab-for-highmem.patch
-dm-larger-dev_t-fix.patch
-rdev-for-samba.patch
-nfsctl-dev_t-fix.patch
-aggregated-disk-stats.patch

 Merged

+irq-printing.patch

 Print friendly info when the new IRQ code detects a problem

+warning-fixes-01.patch
+ipmi-warning-fixes.patch

 Fix some warnings

+irqreturn-i2c.patch
+irqreturn-usb.patch
+irqreturn-uml.patch
+irqreturn-sound-2.patch
+irqreturn-smcc.patch
+irqreturn-aic79xx.patch

 More IRQ fixes

+devfs-brown-bag.patch

 devfs fix

+eisa-sysfs-update.patch

 EISA update

+devfs-pty-fix.patch
+devfs-partition-fix.patch

 More devfs fixes

+SLAB_NO_GROW-fix.patch

 Fix slab/memory allocation interaction

+kgdb-ga-ppc64-fix.patch

 the kgdb patch broke other architectures

+irqreturn-kgdb-ga.patch

 Teach the kgdb stub about the new IRQ API

+kgdb-ga-smp_num_cpus.patch

 smp_num_cpus doesn't exist.

+kgdb-ga-discontigmem-fixup.patch

 Teach the kgdb stub about discontigmem.

+apm-locking-fix.patch

 Locking fix in APM

+ppc64-irqfixes.patch

 Update the pppc64 port for the new IRQ API

+ppc64-pci-bogons.patch

 nail some warnings

+misc.patch

 Minor fixes

+pmd_alloc_one-typo-fix.patch

 Typo fix

+fat-speedup.patch

 Speed up fatfs cluster searching

+cleanups.patch

 Clean up something

+ext3-ro-mount-fix.patch

 Fix ext3 mount-time bug

+nr_threads-docco-fix.patch

 Correct some comments

+lost-tick-HZ-fix.patch

 lost tick detector fixes

+nr_inactive-race-fix.patch

 Fix zone accounting inconsistency in page reclaim

+dcache_lock-vs-tasklist_lock-take-2.patch

 Another attempt at fixing the tasklist_lock/dcache_lock inversion problem

+clone-retval-fix.patch

 Clean up the error returns from clone()

+de_thread-fix.patch

 Fix possible memory corruption in de_thread()

+list_del-debug.patch

 Debugging for list_del()

+airo-schedule-fix.patch

 Don't schedule() in interrupts.

+config-menu-aesthetics.patch

 Config menu cleanup

+aio-retval-fix.patch

 Fix error return value in AIO

+synaptics-mouse-support.patch

 Add support for synaptics inout devices

+pcmcia-20030421.patch

 PCMCIA update (should fix the startup deadlock, but this is not final)

-objrmap.patch

 Is BUGgy

+sched-2.5.68-A9.patch

 HT-aware scheduler

+kgdb-ga-idle-fix.patch

 Update the kgdb stub for the HT-aware scheduler changes

-scheduler-tunables.patch

 It broke again




All 103 patches:


linus.patch

mm.patch
  add -mmN to EXTRAVERSION

irq-printing.patch
  print IRQ handler addresses

warning-fixes-01.patch
  warning fixes

ipmi-warning-fixes.patch

irqreturn-i2c.patch

irqreturn-usb.patch

irqreturn-uml.patch
  UML updates for the new IRQ API

irqreturn-sound-2.patch
  irqs: sound fixups

irqreturn-smcc.patch
  IRDA: missing IRQ bits

irqreturn-aic79xx.patch
  Fix aic79xx for new IRQ API

devfs-brown-bag.patch
  2.5.68-bk renames IDE disks, /dev/hda is directory

eisa-sysfs-update.patch
  EISA/sysfs update

devfs-pty-fix.patch

devfs-partition-fix.patch

SLAB_NO_GROW-fix.patch
  Fix slab-vs-gfp bitflag clash

kgdb-ga.patch
  kgdb stub for ia32 (George Anzinger's one)

kgdb-ga-ppc64-fix.patch

irqreturn-kgdb-ga.patch

kgdb-ga-smp_num_cpus.patch

kgdb-ga-discontigmem-fixup.patch
  kgdb: discontigmem fixup

apm-locking-fix.patch
  APM locking fix

kobj_lock-fix.patch

ppa-null-pointer-fix.patch

config_spinline.patch
  uninline spinlocks for profiling accuracy.

ppc64-reloc_hide.patch

ppc64-pci-patch.patch
  Subject: pci patch

ppc64-aio-32bit-emulation.patch
  32/64bit emulation for aio

ppc64-scruffiness.patch
  Fix some PPC64 compile warnings

ppc64-update.patch
  ppc64 update

ppc64-update-fixes.patch

ppc64-irqfixes.patch

ppc64-pci-bogons.patch

sym-do-160.patch
  make the SYM driver do 160 MB/sec

misc.patch

pmd_alloc_one-typo-fix.patch
  fix typo in m68k mm code

config-PAGE_OFFSET.patch
  Configurable kenrel/user memory split

fat-speedup.patch
  From: Bjorn Stenberg <bjorn@haxx.se>
  fat cluster search speedup

buffer-debug.patch
  buffer.c debugging

ext3-truncate-ordered-pages.patch
  ext3: explicitly free truncated pages

reiserfs_file_write-5.patch

cleanups.patch
  misc cleanups

sched_idle-typo-fix.patch
  fix sched_idle typo

rcu-stats.patch
  RCU statistics reporting

ext3-journalled-data-assertion-fix.patch
  Remove incorrect assertion from ext3

nfs-speedup.patch

nfs-oom-fix.patch
  nfs oom fix

sk-allocation.patch
  Subject: Re: nfs oom

nfs-more-oom-fix.patch

rpciod-atomic-allocations.patch
  Make rcpiod use atomic allocations

linux-isp.patch

isp-update-1.patch

ext3-ro-mount-fix.patch
  fs/ext3/super.c fix for orphan recovery error path

nr_threads-docco-fix.patch
  update nr_threads commentary

lost-tick-HZ-fix.patch
  lost_tick fixes

nr_inactive-race-fix.patch
  zone accounting race fix

dcache_lock-vs-tasklist_lock-take-2.patch
  Fix dcache_lock/tasklist_lock ranking bug

clone-retval-fix.patch
  copy_process return value fix

de_thread-fix.patch
  de_thread memory corruption fix

list_del-debug.patch
  list_del debug check

airo-schedule-fix.patch
  airo.c: don't sleep in atomic regions

config-menu-aesthetics.patch
  config menu: a combination of aesthetics

aio-retval-fix.patch
  aio: fix sys_io_setup error return value

synaptics-mouse-support.patch
  Add Synaptics touchpad tweaking to psmouse driver

kblockd.patch
  Create `kblockd' workqueue

cfq-infrastructure.patch

elevator-completion-api.patch
  elevator completion API

as-iosched.patch
  anticipatory I/O scheduler

as-use-completion.patch
  AS use completion notifier

unplug-use-kblockd.patch
  Use kblockd for running request queues

cfq-2.patch
  CFQ scheduler, #2

unmap-page-debugging.patch
  unmap unused pages for debugging

unmap-page-debugging-fixes.patch

global_flush_tlb-irqs-check.patch

unmap-page-debugging-fixes-2.patch

pcmcia-20030421.patch

fremap-all-mappings.patch
  Make all executable mappings be nonlinear

sched-2.5.68-A9.patch
  HT scheduler, sched-2.5.68-A9

kgdb-ga-idle-fix.patch

sched-2.5.64-D3.patch
  sched-2.5.64-D3, more interactivity changes

show_task-free-stack-fix.patch
  show_task() fix and cleanup

generic-bitops-update.patch
  include/asm-generic/bitops.h {set,clear}_bit return  void

htree-nfs-fix.patch
  Fix ext3 htree / NFS compatibility problems

i8042-share-irqs.patch
  allow i8042 interrupt sharing

select-speedup.patch
  Subject: Re: IA64 changes to fs/select.c

select-speedup-fix.patch
  select() sleedup fix

slab_store_user-large-objects.patch
  slab debug: perform redzoning against larger objects

htree-nfs-fix-2.patch
  htree nfs fix

htree-leak-fix.patch
  ext3: htree memory leak fix

put_task_struct-debug.patch

ia32-mknod64.patch
  mknod64 for ia32

ext2-64-bit-special-inodes.patch
  ext2: support for 64-bit device nodes

ext3-64-bit-special-inodes.patch
  ext3: support for 64-bit device nodes

64-bit-dev_t-kdev_t.patch
  64-bit dev_t and kdev_t

oops-dump-preceding-code.patch
  i386 oops output: dump preceding code

lockmeter.patch

ext3-no-bkl.patch

journal_dirty_metadata-speedup.patch

journal_get_write_access-speedup.patch

ext3-concurrent-block-inode-allocation.patch
  Subject: [PATCH] concurrent block/inode allocation for EXT3

ext3-orlov-approx-counter-fix.patch
  Fix orlov allocator boundary case

ext3-concurrent-block-allocation-fix-1.patch

ext3-concurrent-block-allocation-hashed.patch
  Subject: Re: [PATCH] concurrent block/inode allocation for EXT3

Re: 2.5.68-mm2

[William Lee Irwin III]

On Wed, Apr 23, 2003 at 01:20:46AM -0700, Andrew Morton wrote:
> http://www.zip.com.au/~akpm/linux/patches/2.5/2.5.68/2.5.68-mm2.gz
>    Will appear sometime at:
> ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.5/2.5.68/2.5.68-mm2/
> . Zillions of new fixes.
> . I got tired of the objrmap code going BUG under stress, so it is now in
>   disgrace in the experimental/ directory.

rml and I coordinated to put together a small patch (combining both
our own) for properly locking the static variables in out_of_memory().
There's not any evidence things are going wrong here now, but it at
least addresses the visible lack of locking in out_of_memory().

Applies cleanly to 2.5.68-mm2.


-- wli


diff -urpN mm1-2.5.68-1/mm/oom_kill.c mm1-2.5.68-1A/mm/oom_kill.c
--- mm1-2.5.68-1/mm/oom_kill.c	2003-04-20 00:24:46.000000000 -0700
+++ mm1-2.5.68-1A/mm/oom_kill.c	2003-04-22 21:43:40.000000000 -0700
@@ -208,6 +208,11 @@ static void oom_kill(void)
  */
 void out_of_memory(void)
 {
+	/*
+	 * oom_lock protects out_of_memory()'s static variables.
+	 * It's a global lock; this is not performance-critical.
+	 */
+	static spinlock_t oom_lock = SPIN_LOCK_UNLOCKED;
 	static unsigned long first, last, count, lastkill;
 	unsigned long now, since;
 
@@ -217,6 +222,7 @@ void out_of_memory(void)
 	if (nr_swap_pages > 0)
 		return;
 
+	spin_lock(&oom_lock);
 	now = jiffies;
 	since = now - last;
 	last = now;
@@ -235,14 +241,14 @@ void out_of_memory(void)
 	 */
 	since = now - first;
 	if (since < HZ)
-		return;
+		goto out_unlock;
 
 	/*
 	 * If we have gotten only a few failures,
 	 * we're not really oom. 
 	 */
 	if (++count < 10)
-		return;
+		goto out_unlock;
 
 	/*
 	 * If we just killed a process, wait a while
@@ -251,15 +257,27 @@ void out_of_memory(void)
 	 */
 	since = now - lastkill;
 	if (since < HZ*5)
-		return;
+		goto out_unlock;
 
 	/*
 	 * Ok, really out of memory. Kill something.
 	 */
 	lastkill = now;
+
+	/* oom_kill() sleeps */
+	spin_unlock(&oom_lock);
 	oom_kill();
+	spin_lock(&oom_lock);
 
 reset:
-	first = now;
+	/*
+	 * We dropped the lock above, so check to be sure the variable
+	 * first only ever increases to prevent false OOM's.
+	 */
+	if (time_after(now, first))
+		first = now;
 	count = 0;
+
+out_unlock:
+	spin_unlock(&oom_lock);
 }

Re: 2.5.68-mm2

[Robert Love]

On Wed, 2003-04-23 at 05:59, William Lee Irwin III wrote:

> rml and I coordinated to put together a small patch (combining both
> our own) for properly locking the static variables in out_of_memory().
> There's not any evidence things are going wrong here now, but it at
> least addresses the visible lack of locking in out_of_memory().

Thank you for posting this, wli.

> -	first = now;
> +	/*
> +	 * We dropped the lock above, so check to be sure the variable
> +	 * first only ever increases to prevent false OOM's.
> +	 */
> +	if (time_after(now, first))
> +		first = now;

Just thinking... this little bit is actually a bug even on UP sans
kernel preemption, too, since oom_kill() can sleep.  If it sleeps, and
another process enters out_of_memory(), 'now' and 'first' will be out of
sync.

So I think this patch is a Good Thing in more ways than the obvious SMP
or kernel preemption issue.

	Robert Love

Re: 2.5.68-mm2

[Martin J. Bligh]

>> rml and I coordinated to put together a small patch (combining both
>> our own) for properly locking the static variables in out_of_memory().
>> There's not any evidence things are going wrong here now, but it at
>> least addresses the visible lack of locking in out_of_memory().
> 
> Thank you for posting this, wli.
> 
>> -	first = now;
>> +	/*
>> +	 * We dropped the lock above, so check to be sure the variable
>> +	 * first only ever increases to prevent false OOM's.
>> +	 */
>> +	if (time_after(now, first))
>> +		first = now;
> 
> Just thinking... this little bit is actually a bug even on UP sans
> kernel preemption, too, since oom_kill() can sleep.  If it sleeps, and
> another process enters out_of_memory(), 'now' and 'first' will be out of
> sync.
> 
> So I think this patch is a Good Thing in more ways than the obvious SMP
> or kernel preemption issue.

Is this the bug that akpm was seeing, or a different one? The only 
information I've seen (indirectly) is that fsx triggers the oops.

M.

Re: 2.5.68-mm2

[Robert Love]

On Wed, 2003-04-23 at 12:57, Martin J. Bligh wrote:

> Is this the bug that akpm was seeing, or a different one? The only 
> information I've seen (indirectly) is that fsx triggers the oops.

I cannot see this cause an oops, so no.

Just out-of-sync values resulting in an unexpected OOM or a delayed OOM.

	Robert Love

Re: 2.5.68-mm2

[William Lee Irwin III]

On Wed, Apr 23, 2003 at 02:59:26AM -0700, William Lee Irwin III wrote:
> rml and I coordinated to put together a small patch (combining both
> our own) for properly locking the static variables in out_of_memory().
> There's not any evidence things are going wrong here now, but it at
> least addresses the visible lack of locking in out_of_memory().
> Applies cleanly to 2.5.68-mm2.

Improved OOM killer behavior verified on 64GB i386.


-- wli

Re: 2.5.68-mm2

[Martin J. Bligh]

> . I got tired of the objrmap code going BUG under stress, so it is now in
>   disgrace in the experimental/ directory.

Any chance of some more info on that? BUG at what point in the code,
and with what test to reproduce?

M.

Re: 2.5.68-mm2

[Alex Tomas]

>>>>> Martin J Bligh (MJB) writes:

 >> . I got tired of the objrmap code going BUG under stress, so it is now in
 >> disgrace in the experimental/ directory.

 MJB> Any chance of some more info on that? BUG at what point in the code,
 MJB> and with what test to reproduce?

I've seen this running fsx-linux on ext3

Re: 2.5.68-mm2

[Andrew Morton]

"Martin J. Bligh" <mbligh@aracnet.com> wrote:
>
> > . I got tired of the objrmap code going BUG under stress, so it is now in
> >   disgrace in the experimental/ directory.
> 
> Any chance of some more info on that? BUG at what point in the code,
> and with what test to reproduce?

A bash-shared-mapping (from ext3 CVS) will quickly knock it over.  It gets
its PageAnon/page->mapping state tangled up.

Must confess that I have trouble getting excited over objrmap.  It introduces

- inconsistency (pte_chains versus vma-list scanning)

- code complexity

- a quadratic search

- nasty, nasty problems with remap_file_pages().  I'd rather not have to
  nobble remap_file_pages() functionality for this reason.

and what do we gain from it all?  The small fork/exec boost isn't very
significant.  What we gain is more lowmem space on
going-away-real-soon-now-we-sincerely-hope highmem boxes.

Ingo-rmap seems a better solution to me.  It would be a fairly large change
though - we'd have to hold the four atomic kmaps across an entire pte page
in copy_page_range(), for example.  But it will then have good locality of
reference between adjacent pages and may well be quicker than pte_chains.

Re: 2.5.68-mm2

[Martin J. Bligh]

>> > . I got tired of the objrmap code going BUG under stress, so it is now in
>> >   disgrace in the experimental/ directory.
>> 
>> Any chance of some more info on that? BUG at what point in the code,
>> and with what test to reproduce?
> 
> A bash-shared-mapping (from ext3 CVS) will quickly knock it over.  It gets
> its PageAnon/page->mapping state tangled up.

OK, will try to reproduce that.
 
> - nasty, nasty problems with remap_file_pages().  I'd rather not have to
>   nobble remap_file_pages() functionality for this reason.

I don't see having to predeclare the thing as non-linear as a serious 
imposition .... I don't think memlocking them is necessary, AFAICS if
we have that.
 
> and what do we gain from it all?  The small fork/exec boost isn't very
> significant.  What we gain is more lowmem space on
> going-away-real-soon-now-we-sincerely-hope highmem boxes.

They're not going away soon (unfortunately) - even if Intel stopped producing
the chips today, the machines based on them are still in the marketplace for
years.

The performance improvement was about 25% of systime according to my 
measurements - I don't call that insignificant.

> Ingo-rmap seems a better solution to me.  It would be a fairly large change
> though - we'd have to hold the four atomic kmaps across an entire pte page
> in copy_page_range(), for example.  But it will then have good locality of
> reference between adjacent pages and may well be quicker than pte_chains.

If there was an existing implementation we could actually measure, I'd
be more impressed. From what I can see currently, it'll just introduce
masses of kmap thrashing crap with no obvious way to fix it. And it 
triples the PTE overhead. Maybe it'd work better in conjunction with 
shared pagetables.

M.

Re: 2.5.68-mm2

[Benjamin LaHaise]

On Wed, Apr 23, 2003 at 02:47:32PM -0700, Martin J. Bligh wrote:
> The performance improvement was about 25% of systime according to my 
> measurements - I don't call that insignificant.

Never, ever use changes in system time as a justification for a patch.  We 
all know that Linux's user/system time accounting is patently unreliable.  
Remember Nyquist?  Talk to me about differences in wall clock and your 
comments will be more interesting.

		-ben

Re: 2.5.68-mm2

[Martin J. Bligh]

>> The performance improvement was about 25% of systime according to my 
>> measurements - I don't call that insignificant.
> 
> Never, ever use changes in system time as a justification for a patch.  We 
> all know that Linux's user/system time accounting is patently unreliable.  

Mmmm. I'm not particularly convinced by that ... I do 5 runs for every 
benchmark and compare the results, and it seems very consistent to me. 
For kernbench, it's interesting to look at system time - but obviously
keeping an eye on elapsed time as well, particularly for things like
scheduler patches.

> Remember Nyquist?  Talk to me about differences in wall clock and your 
> comments will be more interesting.

OK, well then you need to look at something that's not totally dominated
by gcc anyway. I know everyone hates SDET as it's "closed" but I'll try
to rerun with aim7 at some point. A real 20% improvement in throughput
is not to be sniffed at ...

DISCLAIMER: SPEC(tm) and the benchmark name SDET(tm) are registered
trademarks of the Standard Performance Evaluation Corporation. This 
benchmarking was performed for research purposes only, and the run results
are non-compliant and not-comparable with any published results.

Results are shown as percentages of the first set displayed

SDET 1  (see disclaimer)
                           Throughput    Std. Dev
                   2.5.68       100.0%         0.7%
           2.5.68-objrmap       105.7%         0.4%

SDET 2  (see disclaimer)
                           Throughput    Std. Dev
                   2.5.68       100.0%         2.8%
           2.5.68-objrmap       108.2%         0.7%

SDET 4  (see disclaimer)
                           Throughput    Std. Dev
                   2.5.68       100.0%         1.0%
           2.5.68-objrmap       112.0%         1.4%

SDET 8  (see disclaimer)
                           Throughput    Std. Dev
                   2.5.68       100.0%         0.6%
           2.5.68-objrmap       122.8%         1.3%

SDET 16  (see disclaimer)
                           Throughput    Std. Dev
                   2.5.68       100.0%         0.1%
           2.5.68-objrmap       117.3%         0.8%

SDET 32  (see disclaimer)
                           Throughput    Std. Dev
                   2.5.68       100.0%         0.4%
           2.5.68-objrmap       118.5%         0.4%

SDET 64  (see disclaimer)
                           Throughput    Std. Dev
                   2.5.68       100.0%         0.2%
           2.5.68-objrmap       121.2%         0.3%

SDET 128  (see disclaimer)
                           Throughput    Std. Dev
                   2.5.68       100.0%         0.1%
           2.5.68-objrmap       118.6%         0.2%

Re: objrmap (was 2.5.68-mm2)

[Martin J. Bligh]

> OK, well then you need to look at something that's not totally dominated
> by gcc anyway. I know everyone hates SDET as it's "closed" but I'll try
> to rerun with aim7 at some point. A real 20% improvement in throughput
> is not to be sniffed at ...

BTW, if you want to see the profile for this, it's obvious what's
taking the time ...

86159 page_remove_rmap
38690 page_add_rmap
17976 zap_pte_range
14431 copy_page_range
10953 __d_lookup
9978 release_pages
9369 find_get_page
7483 atomic_dec_and_lock
6924 __copy_to_user_ll
6830 kmem_cache_free
5848 path_lookup
4687 follow_mount
4430 clear_page_tables
4214 remove_shared_vm_struct
3907 do_wp_page
3823 .text.lock.dec_and_lock
3336 do_no_page
3315 do_anonymous_page
3294 copy_mm
3279 free_pages_and_swap_cache
3111 pte_alloc_one
2709 .text.lock.dcache
2625 .text.lock.filemap
2573 filemap_nopage
2564 copy_process
2556 proc_pid_stat
2358 link_path_walk
2246 do_page_fault
2202 file_move
2189 buffered_rmqueue
2141 free_hot_cold_page
2140 schedule
2114 path_release
1879 current_kernel_time
1825 .text.lock.namei
1722 d_alloc
1719 release_task
1490 __set_page_dirty_buffers
1464 number
1350 kmalloc
1343 __read_lock_failed
1305 page_address
1286 fd_install
1255 __find_get_block
1253 flush_signal_handlers
1249 __fput
1248 exit_notify
1242 task_mem
1221 grab_block
1188 .text.lock.highmem
1169 __block_prepare_write
1123 __brelse
1050 file_kill
1026 .text.lock.file_table
1013 ext2_new_inode
1008 __mark_inode_dirty

Re: 2.5.68-mm2

[Benjamin LaHaise]

On Wed, Apr 23, 2003 at 02:46:48PM -0700, Andrew Morton wrote:
> Ingo-rmap seems a better solution to me.  It would be a fairly large change
> though - we'd have to hold the four atomic kmaps across an entire pte page
> in copy_page_range(), for example.  But it will then have good locality of
> reference between adjacent pages and may well be quicker than pte_chains.

Actually, Ingo's rmap style sounds very similar to what I first implemented 
in one of my stabs at rmap.  It has a nasty side effect of being worst case 
for cache organisation -- the sister page tends to map to the exact same 
cache line in some processors.  Whoops.  That said, I think that the rmap 
pte-chains can really stand a bit of optimization by means of discarding a 
couple of bits, as well as merging for adjacent pages, so I don't think 
the overhead is a lost cause yet.  And nobody has written the clone() patch 
for bash yet...

		-ben
-- 
Junk email?  <a href="mailto:aart@kvack.org">aart@kvack.org</a>

Re: 2.5.68-mm2

[Bill Davidsen]

On Wed, 23 Apr 2003, Benjamin LaHaise wrote:

> Actually, Ingo's rmap style sounds very similar to what I first implemented 
> in one of my stabs at rmap.  It has a nasty side effect of being worst case 
> for cache organisation -- the sister page tends to map to the exact same 
> cache line in some processors.  Whoops.  That said, I think that the rmap 
> pte-chains can really stand a bit of optimization by means of discarding a 
> couple of bits, as well as merging for adjacent pages, so I don't think 
> the overhead is a lost cause yet.  And nobody has written the clone() patch 
> for bash yet...

I'm not sure the best solution is to try to hack applications doing things
in the way they find best. I suspect that we have to change the kernel so
it handles the requests in a reasonable way.

Of course reasonable way may mean that bash does some things a bit slower,
but given that the whole thing works well in most cases anyway, I think
the kernel handling the situation is preferable.

-- 
bill davidsen <davidsen@tmr.com>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.

Re: 2.5.68-mm2

[Benjamin LaHaise]

On Thu, Apr 24, 2003 at 04:24:56PM -0400, Bill Davidsen wrote:
> Of course reasonable way may mean that bash does some things a bit slower,
> but given that the whole thing works well in most cases anyway, I think
> the kernel handling the situation is preferable.

Eh?  It makes bash _faster_ for all cases of starting up a child process.  
And it even works on 2.4 kernels.

		-ben
-- 
Junk email?  <a href="mailto:aart@kvack.org">aart@kvack.org</a>

Re: 2.5.68-mm2

[Bill Davidsen]

On Thu, 24 Apr 2003, Benjamin LaHaise wrote:

> On Thu, Apr 24, 2003 at 04:24:56PM -0400, Bill Davidsen wrote:
> > Of course reasonable way may mean that bash does some things a bit slower,
> > but given that the whole thing works well in most cases anyway, I think
> > the kernel handling the situation is preferable.
> 
> Eh?  It makes bash _faster_ for all cases of starting up a child process.  
> And it even works on 2.4 kernels.

The point is that even if bash is fixed it's desirable to address the
issue in the kernel, other applications may well misbehave as well.

-- 
bill davidsen <davidsen@tmr.com>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.

Re: 2.5.68-mm2

[Randy.Dunlap]

On Fri, 25 Apr 2003 13:56:31 -0400 (EDT) Bill Davidsen <davidsen@tmr.com> wrote:

| On Thu, 24 Apr 2003, Benjamin LaHaise wrote:
| 
| > On Thu, Apr 24, 2003 at 04:24:56PM -0400, Bill Davidsen wrote:
| > > Of course reasonable way may mean that bash does some things a bit slower,
| > > but given that the whole thing works well in most cases anyway, I think
| > > the kernel handling the situation is preferable.
| > 
| > Eh?  It makes bash _faster_ for all cases of starting up a child process.  
| > And it even works on 2.4 kernels.
| 
| The point is that even if bash is fixed it's desirable to address the
| issue in the kernel, other applications may well misbehave as well.

So when would this ever end?

--
~Randy

Re: 2.5.68-mm2

[Robert Love]

On Fri, 2003-04-25 at 14:20, Randy.Dunlap wrote:
>  
> | The point is that even if bash is fixed it's desirable to address the
> | issue in the kernel, other applications may well misbehave as well.
> 
> So when would this ever end?

Exactly what I was thinking.

The kernel cannot be expected to cater to applications or make
concessions (read: hacks) for certain behavior.  If we offer a cleaner,
improved interface which offers the performance improvement, we are
done.  Applications need to start using it.

Of course, I am not arguing against optimizing the old interfaces or
anything of that nature.  I just believe we should not introduce hacks
for application behavior.  It is their job to do the right thing.

	Robert Love

Re: 2.5.68-mm2

[Martin J. Bligh]

>> | The point is that even if bash is fixed it's desirable to address the
>> | issue in the kernel, other applications may well misbehave as well.
>> 
>> So when would this ever end?
> 
> Exactly what I was thinking.
> 
> The kernel cannot be expected to cater to applications or make
> concessions (read: hacks) for certain behavior.  If we offer a cleaner,
> improved interface which offers the performance improvement, we are
> done.  Applications need to start using it.
> 
> Of course, I am not arguing against optimizing the old interfaces or
> anything of that nature.  I just believe we should not introduce hacks
> for application behavior.  It is their job to do the right thing.

I would actually like us to do this (the non-deterministic nature of UNIX
semantics wrt exec is hateful), but changing the kernel before the apps is
ass-backwards. Once this distros fix all their binaries (at least in their
bleeding edge versions) this makes more sense.

There are also some interesting comments the manpage for vfork:

BUGS
       It is rather unfortunate that Linux revived  this  spectre
       from  the past.  The BSD manpage states: "This system call
       will be eliminated when proper system  sharing  mechanisms
       are  implemented.  Users  should  not depend on the memory
       sharing semantics of vfork as it will, in  that  case,  be
       made synonymous to fork."

       Formally  speaking,  the  standard description given above
       does not allow one to use vfork() since a  following  exec
       might fail, and then what happens is undefined.

       Details  of  the  signal  handling  are obscure and differ
       between systems.  The BSD manpage states: "To avoid a pos­
       sible  deadlock  situation, processes that are children in
       the middle of a vfork are never sent  SIGTTOU  or  SIGTTIN
       signals;  rather,  output  or ioctls are allowed and input
       attempts result in an end-of-file indication."

       Currently (Linux 2.3.25), strace(1) cannot follow  vfork()
       and requires a kernel patch.

Re: 2.5.68-mm2

[Bill Davidsen]

On 25 Apr 2003, Robert Love wrote:

> On Fri, 2003-04-25 at 14:20, Randy.Dunlap wrote:
> >  
> > | The point is that even if bash is fixed it's desirable to address the
> > | issue in the kernel, other applications may well misbehave as well.
> > 
> > So when would this ever end?
> 
> Exactly what I was thinking.
> 
> The kernel cannot be expected to cater to applications or make
> concessions (read: hacks) for certain behavior.  If we offer a cleaner,
> improved interface which offers the performance improvement, we are
> done.  Applications need to start using it.
> 
> Of course, I am not arguing against optimizing the old interfaces or
> anything of that nature.  I just believe we should not introduce hacks
> for application behavior.  It is their job to do the right thing.

I don't care much if the kernel does something to make an application run
better, that's an application problem. But if an application can do
something which hurts the performance of the system as a whole, then the
kernel should protect itself and the rest of the system.

So I'm not advocating that the kernel cater to bash, just that doing
legitimate things with bash not have a disproportionate impact on the rest
of the system.

-- 
bill davidsen <davidsen@tmr.com>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.

Re: 2.5.68-mm2

[Martin J. Bligh]

>> > | The point is that even if bash is fixed it's desirable to address the
>> > | issue in the kernel, other applications may well misbehave as well.
>> > 
>> > So when would this ever end?
>> 
>> Exactly what I was thinking.
>> 
>> The kernel cannot be expected to cater to applications or make
>> concessions (read: hacks) for certain behavior.  If we offer a cleaner,
>> improved interface which offers the performance improvement, we are
>> done.  Applications need to start using it.
>> 
>> Of course, I am not arguing against optimizing the old interfaces or
>> anything of that nature.  I just believe we should not introduce hacks
>> for application behavior.  It is their job to do the right thing.
> 
> I don't care much if the kernel does something to make an application run
> better, that's an application problem. But if an application can do
> something which hurts the performance of the system as a whole, then the
> kernel should protect itself and the rest of the system.
> 
> So I'm not advocating that the kernel cater to bash, just that doing
> legitimate things with bash not have a disproportionate impact on the rest
> of the system.

It's not just bash ... it's most applications.

M.

[BUG] 2.5.68-mm2 and list.h

[Alexander Hoogerhuis]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Andrew Morton <akpm@digeo.com> writes:
>
> [SNIP]
>

Caught this one on boot:

drivers/usb/host/uhci-hcd.c: USB Universal Host Controller Interface driver v2.0
Bluetooth: Core ver 2.2
Bluetooth: HCI device and connection manager initialized
Bluetooth: HCI socket layer initialized
Bluetooth: HCI USB driver ver 2.1
drivers/usb/core/usb.c: registered new driver hci_usb
drivers/usb/core/message.c: usb_control/bulk_msg: timeout
- ------------[ cut here ]------------
kernel BUG at include/linux/list.h:140!
invalid operand: 0000 [#1]
CPU:    0
EIP:    0060:[<c011acc2>]    Not tainted VLI
EFLAGS: 00010083
EIP is at remove_wait_queue+0x66/0x70
eax: ec937d94   ebx: ec936000   ecx: ec937da0   edx: ec797d28
esi: 00000286   edi: ec937d94   ebp: ec937d58   esp: ec937d50
ds: 007b   es: 007b   ss: 0068
Process logger (pid: 3942, threadinfo=ec936000 task=edede700)
Stack: ec797d28 ec936000 ec937dc0 c019e462 c02fff00 00000002 c0117b14 ec797d24
       effb1600 00000000 edede700 c0119716 00000000 00000000 00000000 ec937ee7
       ecaa5df0 00000000 edede700 c0119716 ec797d28 ec797d28 ec937ee7 0026c8ca
Call Trace:
 [<c019e462>] devfs_d_revalidate_wait+0x181/0x18d
 [<c0117b14>] do_page_fault+0x0/0x4bf
 [<c0119716>] default_wake_function+0x0/0x12
 [<c0119716>] default_wake_function+0x0/0x12
 [<c015add3>] do_lookup+0x5c/0x98
 [<c015b2c6>] link_path_walk+0x4b7/0x8fd
 [<c01350cd>] file_read_actor+0x0/0x11f
 [<c01350cd>] file_read_actor+0x0/0x11f
 [<c029d529>] unix_find_other+0x2e/0x158
 [<c029dac4>] unix_dgram_connect+0xfb/0x1a5
 [<c024a9ec>] sys_connect+0xa9/0xb1
 [<c024953d>] sock_map_fd+0x118/0x12e
 [<c024a586>] sys_socket+0x3a/0x56
 [<c024b50f>] sys_socketcall+0xb2/0x262
 [<c015ef49>] do_fcntl+0xd8/0x1a4
 [<c015f0fa>] sys_fcntl64+0x6f/0xab
 [<c010ae57>] syscall_call+0x7/0xb
                                                                                                                                                                                                                                    
Code: 43 08 83 e0 08 75 0b 8b 1c 24 8b 74 24 04 89 ec 5d c3 8b 1c 24 8b 74 24 04 89 ec 5d e9 0e ea ff ff 0f 0b 8d 00 12 42 2b c0 eb c9 <0f> 0b 8c 00 12 42 2b c0 eb b7 55 89 e5 83 ec 0c 89 1c 24 89 74
 <6>note: logger[3942] exited with preempt_count 1
bad: scheduling while atomic!
Call Trace:
 [<c01196c1>] schedule+0x3f1/0x3f6
 [<c013f9f6>] unmap_page_range+0x41/0x67
 [<c013fbd1>] unmap_vmas+0x1b5/0x216
 [<c01437da>] exit_mmap+0x7a/0x18d
 [<c010b920>] do_invalid_op+0x0/0xb7
 [<c011b0f2>] mmput+0x67/0xcf
 [<c011ee19>] do_exit+0x159/0x485
 [<c010b920>] do_invalid_op+0x0/0xb7
 [<c010b611>] do_divide_error+0x0/0xde
 [<c010b9d5>] do_invalid_op+0xb5/0xb7
 [<c011acc2>] remove_wait_queue+0x66/0x70
 [<c0117d92>] do_page_fault+0x27e/0x4bf
 [<c010b001>] error_code+0x2d/0x38
 [<c011acc2>] remove_wait_queue+0x66/0x70
 [<c019e462>] devfs_d_revalidate_wait+0x181/0x18d
 [<c0117b14>] do_page_fault+0x0/0x4bf
 [<c0119716>] default_wake_function+0x0/0x12
 [<c0119716>] default_wake_function+0x0/0x12
 [<c015add3>] do_lookup+0x5c/0x98
 [<c015b2c6>] link_path_walk+0x4b7/0x8fd
 [<c01350cd>] file_read_actor+0x0/0x11f
 [<c01350cd>] file_read_actor+0x0/0x11f
 [<c029d529>] unix_find_other+0x2e/0x158
 [<c029dac4>] unix_dgram_connect+0xfb/0x1a5
 [<c024a9ec>] sys_connect+0xa9/0xb1
 [<c024953d>] sock_map_fd+0x118/0x12e
 [<c024a586>] sys_socket+0x3a/0x56
 [<c024b50f>] sys_socketcall+0xb2/0x262
 [<c015ef49>] do_fcntl+0xd8/0x1a4
 [<c015f0fa>] sys_fcntl64+0x6f/0xab
 [<c010ae57>] syscall_call+0x7/0xb

and with modular USB I also get this along for free, right after:

usbfs: USBDEVFS_CONTROL failed dev 2 rqt 128 rq 6 len 9 ret -110
usbfs: USBDEVFS_CONTROL failed dev 2 rqt 128 rq 6 len 18 ret -75
drivers/usb/core/message.c: usb_control/bulk_msg: timeout
usbfs: USBDEVFS_CONTROL failed dev 2 rqt 128 rq 6 len 18 ret -110
usbfs: USBDEVFS_CONTROL failed dev 2 rqt 128 rq 6 len 18 ret -75
drivers/usb/core/message.c: usb_control/bulk_msg: timeout
usbfs: USBDEVFS_CONTROL failed dev 2 rqt 128 rq 6 len 9 ret -110
drivers/usb/core/message.c: usb_control/bulk_msg: timeout
usbfs: USBDEVFS_CONTROL failed dev 2 rqt 128 rq 6 len 18 ret -110
usbfs: USBDEVFS_CONTROL failed dev 2 rqt 128 rq 6 len 9 ret -75
drivers/usb/core/message.c: usb_control/bulk_msg: timeout
usbfs: USBDEVFS_CONTROL failed dev 2 rqt 128 rq 6 len 9 ret -110
usbfs: USBDEVFS_CONTROL failed dev 2 rqt 128 rq 6 len 9 ret -75
drivers/usb/core/message.c: usb_control/bulk_msg: timeout
usbfs: USBDEVFS_CONTROL failed dev 2 rqt 128 rq 6 len 9 ret -110
drivers/usb/core/message.c: usb_control/bulk_msg: timeout
usbfs: USBDEVFS_CONTROL failed dev 2 rqt 128 rq 6 len 9 ret -110
usbfs: USBDEVFS_CONTROL failed dev 2 rqt 128 rq 6 len 9 ret -75
drivers/usb/core/message.c: usb_control/bulk_msg: timeout
usbfs: USBDEVFS_CONTROL failed dev 2 rqt 128 rq 6 len 18 ret -110

.config is attached.

mvh,
A
- -- 
Alexander Hoogerhuis                               | alexh@ihatent.com
CCNP - CCDP - MCNE - CCSE                          | +47 908 21 485
"You have zero privacy anyway. Get over it."  --Scott McNealy
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.1 (GNU/Linux)
Comment: Processed by Mailcrypt 3.5.8 <http://mailcrypt.sourceforge.net/>

iD8DBQE+sLyBCQ1pa+gRoggRAqPjAJoDxrfbRyPlEzUU0V14WzPhSmPXaQCfThba
/nN9EqynPWOUo+F3T8P6uBE=
=C+e/
-----END PGP SIGNATURE-----

Re: [BUG] 2.5.68-mm2 and list.h

[Andrew Morton]

Alexander Hoogerhuis <alexh@ihatent.com> wrote:
>
> kernel BUG at include/linux/list.h:140!
> Call Trace:
>  [<c019e462>] devfs_d_revalidate_wait+0x181/0x18d

Yes.  Apparently, devfs has some programming flaws.


For now, please just delete the new debug tests in
include/linux/list.h:list_del():

#include <linux/kernel.h>       /* BUG_ON */
static inline void list_del(struct list_head *entry)
{
	BUG_ON(entry->prev->next != entry);
	BUG_ON(entry->next->prev != entry);
	__list_del(entry->prev, entry->next);
}

Those BUG_ON's.