Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Shane Shrybman]

Hi,

Here are some more alsa xrun debug messages with SB Live and onboard via
soundcards, v4l was in use too.

2.6.7-mm7, UP, XP 2500, aic7xxx scsi.

Tried to edit out a lot of the duplicates, the first one appeared very
frequently.

Attached scsi generic sg0 at scsi0, channel 0, id 0, lun 0,  type 0
Attached scsi generic sg1 at scsi0, channel 0, id 1, lun 0,  type 0
ALSA sound/core/pcm_lib.c:169: XRUN: pcmC0D0p
 [<e08ed360>] snd_pcm_period_elapsed+0x2f0/0x3c0 [snd_pcm]
 [<e0915b58>] gcc2_compiled.+0x18/0x20 [snd_emu10k1]
 [<e091415f>] gcc2_compiled.+0xef/0x2e0 [snd_emu10k1]
 [<c0107b51>] handle_IRQ_event+0x31/0x60
 [<c0107ea3>] do_IRQ+0xb3/0x150
 [<c010662c>] common_interrupt+0x18/0x20
ALSA sound/pci/via82xx.c:731: invalid via82xx_cur_ptr, using last valid
pointer
There is already a security framework initialized, register_security
failed.
selinux_register_security:  Registering secondary module realtime
Realtime LSM initialized (all groups, mlock=1)
ALSA sound/core/pcm_lib.c:169: XRUN: pcmC0D0p
 [<e08ed360>] snd_pcm_period_elapsed+0x2f0/0x3c0 [snd_pcm]
 [<e0915b58>] gcc2_compiled.+0x18/0x20 [snd_emu10k1]
 [<e091415f>] gcc2_compiled.+0xef/0x2e0 [snd_emu10k1]
 [<c0107b51>] handle_IRQ_event+0x31/0x60
 [<c0107ea3>] do_IRQ+0xb3/0x150
 [<c0104030>] default_idle+0x0/0x30
 [<c010662c>] common_interrupt+0x18/0x20
 [<c0104030>] default_idle+0x0/0x30
 [<c0104053>] default_idle+0x23/0x30
 [<c01040de>] cpu_idle+0x2e/0x40
 [<c0100387>] rest_init+0x57/0x60
 [<c041c73c>] start_kernel+0x18c/0x1a0
ALSA sound/core/pcm_lib.c:169: XRUN: pcmC0D0c
 [<e08ed360>] snd_pcm_period_elapsed+0x2f0/0x3c0 [snd_pcm]
 [<c0120a27>] __mod_timer+0x157/0x170
 [<e0915b70>] snd_emu10k1_pcm_ac97adc_interrupt+0x10/0x20 [snd_emu10k1]
 [<e09141a7>] gcc2_compiled.+0x137/0x2e0 [snd_emu10k1]
 [<c0107b51>] handle_IRQ_event+0x31/0x60
 [<c0107ea3>] do_IRQ+0xb3/0x150
 [<c0104030>] default_idle+0x0/0x30
 [<c010662c>] common_interrupt+0x18/0x20
 [<c0104030>] default_idle+0x0/0x30
 [<c0104053>] default_idle+0x23/0x30
 [<c01040de>] cpu_idle+0x2e/0x40
 [<c0100387>] rest_init+0x57/0x60
 [<c041c73c>] start_kernel+0x18c/0x1a0
ALSA sound/core/pcm_lib.c:169: XRUN: pcmC0D0p
 [<e08ed360>] snd_pcm_period_elapsed+0x2f0/0x3c0 [snd_pcm]
 [<e0915b58>] gcc2_compiled.+0x18/0x20 [snd_emu10k1]
 [<e091415f>] gcc2_compiled.+0xef/0x2e0 [snd_emu10k1]
 [<c0107b51>] handle_IRQ_event+0x31/0x60
 [<c0107ea3>] do_IRQ+0xb3/0x150
 [<c010662c>] common_interrupt+0x18/0x20

ALSA sound/core/pcm_lib.c:169: XRUN: pcmC0D0p
 [<e08ed360>] snd_pcm_period_elapsed+0x2f0/0x3c0 [snd_pcm]
 [<e0915b58>] gcc2_compiled.+0x18/0x20 [snd_emu10k1]
 [<e091415f>] gcc2_compiled.+0xef/0x2e0 [snd_emu10k1]
 [<c0107b51>] handle_IRQ_event+0x31/0x60
 [<c0107ea3>] do_IRQ+0xb3/0x150
 [<c010662c>] common_interrupt+0x18/0x20
 [<c01d08e4>] fast_clear_page+0x24/0x60
 [<c01d0a66>] mmx_clear_page+0x26/0x30
 [<c013ee13>] do_anonymous_page+0x83/0x180
 [<c013ef61>] do_no_page+0x51/0x320
 [<c011549f>] pte_alloc_one+0x2f/0x40
 [<c013f36d>] handle_mm_fault+0x6d/0x140
 [<c013dfac>] get_user_pages+0x22c/0x3c0
 [<c013f4e9>] make_pages_present+0x69/0x90
 [<c0140d04>] do_mmap_pgoff+0x564/0x650
 [<c010bc46>] sys_mmap2+0x66/0xa0
 [<c0105c6d>] sysenter_past_esp+0x52/0x71


ALSA sound/core/pcm_lib.c:169: XRUN: pcmC0D0p
 [<e08ed360>] snd_pcm_period_elapsed+0x2f0/0x3c0 [snd_pcm]
 [<e0915b58>] gcc2_compiled.+0x18/0x20 [snd_emu10k1]
 [<e091415f>] gcc2_compiled.+0xef/0x2e0 [snd_emu10k1]
 [<c0107b51>] handle_IRQ_event+0x31/0x60
 [<c0107ea3>] do_IRQ+0xb3/0x150
 [<c0104030>] default_idle+0x0/0x30
 [<c010662c>] common_interrupt+0x18/0x20
 [<c0104030>] default_idle+0x0/0x30
 [<c0104053>] default_idle+0x23/0x30
 [<c01040de>] cpu_idle+0x2e/0x40
 [<c0100387>] rest_init+0x57/0x60
 [<c041c73c>] start_kernel+0x18c/0x1a0
ALSA sound/pci/via82xx.c:731: invalid via82xx_cur_ptr, using last valid
pointer
ALSA sound/core/pcm_lib.c:169: XRUN: pcmC0D0p
 [<e08ed360>] snd_pcm_period_elapsed+0x2f0/0x3c0 [snd_pcm]
 [<e0915b58>] gcc2_compiled.+0x18/0x20 [snd_emu10k1]
 [<e091415f>] gcc2_compiled.+0xef/0x2e0 [snd_emu10k1]
 [<c0107b51>] handle_IRQ_event+0x31/0x60
 [<c0107ea3>] do_IRQ+0xb3/0x150
 [<c0104030>] default_idle+0x0/0x30
 [<c010662c>] common_interrupt+0x18/0x20
 [<c0104030>] default_idle+0x0/0x30
 [<c0104053>] default_idle+0x23/0x30
 [<c01040de>] cpu_idle+0x2e/0x40
 [<c0100387>] rest_init+0x57/0x60
 [<c041c73c>] start_kernel+0x18c/0x1a0
ALSA sound/pci/via82xx.c:731: invalid via82xx_cur_ptr, using last valid
pointer
ALSA sound/core/pcm_lib.c:169: XRUN: pcmC0D0p
 [<e08ed360>] snd_pcm_period_elapsed+0x2f0/0x3c0 [snd_pcm]
 [<e0915b58>] gcc2_compiled.+0x18/0x20 [snd_emu10k1]
 [<e091415f>] gcc2_compiled.+0xef/0x2e0 [snd_emu10k1]
 [<c0107b51>] handle_IRQ_event+0x31/0x60
 [<c0107ea3>] do_IRQ+0xb3/0x150
 [<c0104030>] default_idle+0x0/0x30
 [<c010662c>] common_interrupt+0x18/0x20
 [<c0104030>] default_idle+0x0/0x30
 [<c0104053>] default_idle+0x23/0x30
 [<c01040de>] cpu_idle+0x2e/0x40
 [<c0100387>] rest_init+0x57/0x60
 [<c041c73c>] start_kernel+0x18c/0x1a0
ALSA sound/core/pcm_lib.c:169: XRUN: pcmC0D0p
 [<e08ed360>] snd_pcm_period_elapsed+0x2f0/0x3c0 [snd_pcm]
 [<e0915b58>] gcc2_compiled.+0x18/0x20 [snd_emu10k1]
 [<e091415f>] gcc2_compiled.+0xef/0x2e0 [snd_emu10k1]
 [<c0107b51>] handle_IRQ_event+0x31/0x60
 [<c0107ea3>] do_IRQ+0xb3/0x150
 [<c0104030>] default_idle+0x0/0x30
 [<c010662c>] common_interrupt+0x18/0x20
 [<c0104030>] default_idle+0x0/0x30
 [<c0104053>] default_idle+0x23/0x30
 [<c01040de>] cpu_idle+0x2e/0x40
 [<c0100387>] rest_init+0x57/0x60
 [<c041c73c>] start_kernel+0x18c/0x1a0
ALSA sound/core/pcm_lib.c:169: XRUN: pcmC0D0p
 [<e08ed360>] snd_pcm_period_elapsed+0x2f0/0x3c0 [snd_pcm]
 [<e0915b58>] gcc2_compiled.+0x18/0x20 [snd_emu10k1]
 [<e091415f>] gcc2_compiled.+0xef/0x2e0 [snd_emu10k1]
 [<c0107b51>] handle_IRQ_event+0x31/0x60
 [<c0107ea3>] do_IRQ+0xb3/0x150
 [<c010662c>] common_interrupt+0x18/0x20
ALSA sound/core/pcm_lib.c:169: XRUN: pcmC0D0p
 [<e08ed360>] snd_pcm_period_elapsed+0x2f0/0x3c0 [snd_pcm]
 [<e0915b58>] gcc2_compiled.+0x18/0x20 [snd_emu10k1]
 [<e091415f>] gcc2_compiled.+0xef/0x2e0 [snd_emu10k1]
 [<c0107b51>] handle_IRQ_event+0x31/0x60
 [<c0107ea3>] do_IRQ+0xb3/0x150
 [<c0104030>] default_idle+0x0/0x30
 [<c010662c>] common_interrupt+0x18/0x20
 [<c0104030>] default_idle+0x0/0x30
 [<c0104053>] default_idle+0x23/0x30
 [<c01040de>] cpu_idle+0x2e/0x40
 [<c0100387>] rest_init+0x57/0x60
 [<c041c73c>] start_kernel+0x18c/0x1a0

ALSA sound/core/pcm_lib.c:169: XRUN: pcmC0D0p
 [<e08ed360>] snd_pcm_period_elapsed+0x2f0/0x3c0 [snd_pcm]
 [<e0915b58>] gcc2_compiled.+0x18/0x20 [snd_emu10k1]
 [<e091415f>] gcc2_compiled.+0xef/0x2e0 [snd_emu10k1]
 [<c0107b51>] handle_IRQ_event+0x31/0x60
 [<c0107ea3>] do_IRQ+0xb3/0x150
 [<c0104030>] default_idle+0x0/0x30
 [<c010662c>] common_interrupt+0x18/0x20
 [<c0104030>] default_idle+0x0/0x30
 [<c0104053>] default_idle+0x23/0x30
 [<c01040de>] cpu_idle+0x2e/0x40
 [<c0100387>] rest_init+0x57/0x60
 [<c041c73c>] start_kernel+0x18c/0x1a0


ALSA sound/core/pcm_lib.c:169: XRUN: pcmC0D0c
 [<e08ed360>] snd_pcm_period_elapsed+0x2f0/0x3c0 [snd_pcm]
 [<c01169f0>] try_to_wake_up+0x20/0xb0
 [<e0915b70>] snd_emu10k1_pcm_ac97adc_interrupt+0x10/0x20 [snd_emu10k1]
 [<e09141a7>] gcc2_compiled.+0x137/0x2e0 [snd_emu10k1]
 [<c0107b51>] handle_IRQ_event+0x31/0x60
 [<c0107ea3>] do_IRQ+0xb3/0x150
 [<c010662c>] common_interrupt+0x18/0x20
 [<c01d0dca>] __copy_to_user_ll+0x4a/0x70
 [<c01d0e8f>] copy_to_user+0x2f/0x40
 [<c028fb9c>] memcpy_toiovec+0x2c/0x60
 [<c02e34ed>] unix_stream_recvmsg+0x24d/0x3d0
 [<c028a63b>] sock_aio_read+0x11b/0x130
 [<c013505e>] free_pages+0x2e/0x30
 [<c014bb1a>] do_sync_read+0x7a/0xb0
 [<c014bc14>] vfs_read+0xc4/0xf0
 [<c014be11>] sys_read+0x31/0x50
 [<c0105c6d>] sysenter_past_esp+0x52/0x71


ALSA sound/core/pcm_lib.c:169: XRUN: pcmC0D0c
 [<e08ed360>] snd_pcm_period_elapsed+0x2f0/0x3c0 [snd_pcm]
 [<c0116a1f>] try_to_wake_up+0x4f/0xb0
 [<e0915b70>] snd_emu10k1_pcm_ac97adc_interrupt+0x10/0x20 [snd_emu10k1]
 [<e09141a7>] gcc2_compiled.+0x137/0x2e0 [snd_emu10k1]
 [<c0107b51>] handle_IRQ_event+0x31/0x60
 [<c0107ea3>] do_IRQ+0xb3/0x150
 [<c010662c>] common_interrupt+0x18/0x20
 [<c02e32c1>] unix_stream_recvmsg+0x21/0x3d0
 [<c01bc656>] selinux_socket_recvmsg+0x16/0x20
 [<c028a63b>] sock_aio_read+0x11b/0x130
 [<c014bb1a>] do_sync_read+0x7a/0xb0
 [<c014bc14>] vfs_read+0xc4/0xf0
 [<c014be11>] sys_read+0x31/0x50
 [<c0105c6d>] sysenter_past_esp+0x52/0x71

ALSA sound/core/pcm_lib.c:169: XRUN: pcmC0D0c
 [<e08ed360>] snd_pcm_period_elapsed+0x2f0/0x3c0 [snd_pcm]
 [<c0122ca0>] send_group_sig_info+0x20/0x40
 [<e0915b70>] snd_emu10k1_pcm_ac97adc_interrupt+0x10/0x20 [snd_emu10k1]
 [<e09141a7>] gcc2_compiled.+0x137/0x2e0 [snd_emu10k1]
 [<c0107b51>] handle_IRQ_event+0x31/0x60
 [<c0107ea3>] do_IRQ+0xb3/0x150
 [<c010662c>] common_interrupt+0x18/0x20
 [<c013d024>] blk_queue_bounce+0x4/0x50
 [<c0220416>] __make_request+0x46/0x4d0
 [<c02209d2>] generic_make_request+0x132/0x150
 [<c01182d0>] autoremove_wake_function+0x0/0x40
 [<c0220a9b>] submit_bio+0xab/0xc0
 [<c0150508>] bio_alloc+0x128/0x1b0
 [<c014fea8>] submit_bh+0xd8/0x100
 [<c014e1d2>] __bread_slow+0x42/0x80
 [<c014e465>] __bread+0x25/0x30
 [<c0180094>] ext3_get_branch+0x64/0xf0
 [<c0180646>] ext3_get_block_handle+0x96/0x2b0
 [<c0150242>] alloc_buffer_head+0x12/0x50
 [<c01808c0>] ext3_get_block+0x60/0x70
 [<c014eb13>] __block_prepare_write+0x133/0x380
 [<c018d75e>] new_handle+0xe/0x50
 [<c014f481>] block_prepare_write+0x21/0x40
 [<c0180860>] ext3_get_block+0x0/0x70
 [<c0180d29>] ext3_prepare_write+0x59/0x110
 [<c0180860>] ext3_get_block+0x0/0x70
 [<c0132dc8>] generic_file_aio_write_nolock+0x6e8/0xa80
 [<c01ce5ba>] radix_tree_gang_lookup_tag+0x4a/0x60
 [<c01b7aea>] avc_has_perm_noaudit+0x12a/0x1c0
 [<c01b8f49>] inode_has_perm+0x69/0x80
 [<c0133253>] generic_file_aio_write+0x63/0x80
 [<c017eb4b>] ext3_file_write+0x2b/0xc0
 [<c014bcba>] do_sync_write+0x7a/0xb0
 [<c014bda5>] vfs_write+0xb5/0xf0
 [<c014be61>] sys_write+0x31/0x50
 [<c0105c6d>] sysenter_past_esp+0x52/0x71

ALSA sound/core/pcm_lib.c:169: XRUN: pcmC0D0p
 [<e08ed360>] snd_pcm_period_elapsed+0x2f0/0x3c0 [snd_pcm]
 [<e0915b58>] gcc2_compiled.+0x18/0x20 [snd_emu10k1]
 [<e091415f>] gcc2_compiled.+0xef/0x2e0 [snd_emu10k1]
 [<c0107b51>] handle_IRQ_event+0x31/0x60
 [<c0107ea3>] do_IRQ+0xb3/0x150
 [<c010662c>] common_interrupt+0x18/0x20
 [<c01107d1>] get_offset_pmtmr+0x31/0x860
 [<c010b478>] do_gettimeofday+0x18/0xb0
 [<c011d1cc>] sys_gettimeofday+0x1c/0x60
 [<c0105c6d>] sysenter_past_esp+0x52/0x71

ALSA sound/pci/via82xx.c:731: invalid via82xx_cur_ptr, using last valid
pointer

Regards,

Shane

[announce] [patch] Voluntary Kernel Preemption Patch

[Ingo Molnar]

as most of you are probably aware of it, there have been complaints on
lkml that the 2.6 kernel is not suitable for serious audio work due to
high scheduling latencies (e.g. the Jackit people complained). I took a
look at latencies and indeed 2.6.7 is pretty bad - latencies up to 50
msec (!) can be easily triggered using common workloads, on fast 2GHz+
x86 system - even when using the fully preemptible kernel!

to solve this problem, Arjan van de Ven and I went over various kernel
functions to determine their preemptability and we re-created from
scratch a patch that is equivalent in performance to the 2.4 lowlatency
patches but is different in design, impact and approach:

  http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.7-bk20-H2

  (Note to kernel patch reviewers: the split voluntary_resched type of
  APIs, the feature #ifdefs and runtime flags are temporary and were
  only introduced to enable a easy benchmarking/comparisons. I'll split
  this up into small pieces once there's testing feedback and actual
  audio users had their say!)

unlike the lowlatency patches, this patch doesn't add a lot of new
scheduling points to the source code, it rather reuses a rich but
currently inactive set of scheduling points that already exist in the
2.6 tree: the might_sleep() debugging checks. Any code point that does
might_sleep() is in fact ready to sleep at that point. So the patch
activates these debugging checks to be scheduling points. This reduces
complexity and impact quite significantly.

but even using these (over one hundred) might_sleep() points there were
still a number of latency sources in the kernel - we identified and
fixed them by hand, either via additional might_sleep() checks, or via
explicit rescheduling points. Sometimes lock-break was necessary as
well.

as a practical goal, this patch aims to fix all latency sources that
generate higher than ~1 msec latencies. We'd love to learn about
workloads that still cause audio skipping even with this patch applied,
but i've been unable to generate any load that creates higher than 1msec
latencies. (not counting driver initialization routines.)

this patch is also more configurable than the 2.4 lowlatency patches
were: there's a .config option to enable voluntary preemption, and there
are runtime /proc/sys knobs and boot-time flags to turn voluntary
preemption (CONFIG_VOLUNTARY_PREEMPT) and kernel preemption
(CONFIG_PREEMPT) on/off:

        # turn on/off voluntary preemption (if CONFIG_VOLUNTARY_PREEMPT)
	echo 1 > /proc/sys/kernel/voluntary_preemption
	echo 0 > /proc/sys/kernel/voluntary_preemption

        # turn on/off the preemptible kernel feature (if CONFIG_PREEMPT)
	/proc/sys/kernel/kernel_preemption
	/proc/sys/kernel/kernel_preemption

the 'voluntary-preemption=0/1' and 'kernel-preemption=0/1' boot options
can be used to control these flags at boot-time.

all 4 combinations make sense if both CONFIG_PREEMPT and
CONFIG_VOLUNTARY_PREEMPT are enabled - great for performance/latency
testing and comparisons.

The stock 2.6 kernel is equivalent to:

   voluntary_preemption:0 kernel_preemption:0

the 2.6 kernel with voluntary kernel preemption is equivalent to:

   voluntary_preemption:1 kernel_preemption:0

the 2.6 kernel with preemptible kernel enabled is:

   voluntary_preemption:0 kernel_preemption:1

and the preemptible kernel enhanced with additional lock-breaks is 
enabled via:

   voluntary_preemption:1 kernel_preemption:1

it is safe to change these flags anytime.

The patch is against 2.6.7-bk20, and it also includes fixes for kernel
bugs that were uncovered while developing this patch. While it works for
me, be careful when using this patch!

Testreports, comments, suggestions are more than welcome,

	Ingo

Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Andrew Morton]

Ingo Molnar <mingo@elte.hu> wrote:
>
> I took a
>  look at latencies and indeed 2.6.7 is pretty bad - latencies up to 50
>  msec (!) can be easily triggered using common workloads, on fast 2GHz+
>  x86 system - even when using the fully preemptible kernel!

What were those workloads?

Certainly 2.6+preempt is not as good as 2.4+LL at this time, but 2.6 isn't
too bad either.  Even under heavy filesystem load it's hard to exceed a 0.5
millisecond holdoff.  There are still a few problem in the ext3 checkpoint
buffer handling, but those seem pretty hard to hit.  I doubt if the `Jack'
testers were running `dbench 1000' during their testing.  

All of which makes me suspect that the problems which the `Jack' testers
saw were not directly related to long periods of non-preemption in-kernel. 
At least, not in core kernel/fs/mm code.  There have been problem in the
past in places like i2c drivers, fbdev scrolling, etc.

What we need to do is to encourage audio testers to use ALSA drivers, to
enable CONFIG_SND_DEBUG in the kernel build and to set
/proc/asound/*/*/xrun_debug and to send us the traces which result from
underruns.

As for the patch, well, sprinkling rescheduling points everywhere is still
not the preferred approach.  But adding more might_sleep() checks is a
sneaky way of making it more attractive ;)

Minor point:  this:

	cond_resched();
	function_which_might_sleep();

is less efficient than

	function_which_might_sleep();
	cond_resched();

because if function_which_might_sleep() _does_ sleep, need_resched() will
likely be false when we hit cond_resched(), thus saving a context switch. 
Unfortunately, might_sleep() calls tend to go at the entry to functions,
whereas cond_resched() calls should be neat the exit point, or inside loop
bodies.

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Lee Revell]

On Sun, 2004-07-11 at 01:25, Andrew Morton wrote:
> What we need to do is to encourage audio testers to use ALSA drivers, to
> enable CONFIG_SND_DEBUG in the kernel build and to set
> /proc/asound/*/*/xrun_debug and to send us the traces which result from
> underruns.
> 

OK, here goes.  The following traces result from running JACK overnight
like so, on an otherwise idle system.  Hardware is a VIA EPIA 6000, with
a 600Mhz C3 processor.  Kernel is 2.6.7 + volunatary_preempt patch. 
voluntary_preempt and kernel_preemption are both on.

jackd -v --realtime -d alsa --outchannels 2 --rate 48000 --shorts
--playback --period 32  --nperiods 2

These settings require less than 666 microseconds scheduler latency. 
The average performance is quite good - 5-20 *microseconds*!

load = 0.3754 max usecs: 5.000, spare = 661.000
load = 1.1637 max usecs: 13.000, spare = 653.000
load = 3.0593 max usecs: 33.000, spare = 633.000
load = 1.9050 max usecs: 5.000, spare = 661.000
load = 1.4780 max usecs: 7.000, spare = 659.000
load = 1.2645 max usecs: 7.000, spare = 659.000
load = 1.0076 max usecs: 5.000, spare = 661.000
load = 1.1044 max usecs: 8.000, spare = 658.000
load = 0.9276 max usecs: 5.000, spare = 661.000
load = 1.5148 max usecs: 14.000, spare = 652.000
load = 1.1328 max usecs: 5.000, spare = 661.000
load = 1.0168 max usecs: 6.000, spare = 660.000

However there are periodic XRUNS.  I noticed that many of these seem
APM-related, so here are the relevant settings:

# Power management options (ACPI, APM)
# APM (Advanced Power Management) BIOS Support
CONFIG_APM=y
# CONFIG_APM_IGNORE_USER_SUSPEND is not set
# CONFIG_APM_DO_ENABLE is not set
CONFIG_APM_CPU_IDLE=y
# CONFIG_APM_DISPLAY_BLANK is not set
CONFIG_APM_RTC_IS_GMT=y
# CONFIG_APM_ALLOW_INTS is not set
# CONFIG_APM_REAL_MODE_POWER_OFF is not set

I did not intend to activate these settings when I compiled the kernel,
but as this is certainly not an idle system, it seems like there may be
a bug lurking.

The log excerpts below are annotated.  Sorry if it is too long.

This section of the log is from running JACK overnight:

Jul 12 06:56:07 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 06:56:07 mindpipe kernel:  [__crc_totalram_pages+1387264/5353478] snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 06:56:07 mindpipe kernel:  [__crc_totalram_pages+1455070/5353478] snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 06:56:07 mindpipe kernel:  [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 06:56:07 mindpipe kernel:  [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 06:56:07 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 06:56:07 mindpipe kernel:  [preempt_schedule+16/80] preempt_schedule+0x10/0x50
Jul 12 06:56:07 mindpipe kernel:  [rt_run_flush+135/144] rt_run_flush+0x87/0x90
Jul 12 06:56:07 mindpipe kernel:  [rt_secret_rebuild+0/48] rt_secret_rebuild+0x0/0x30
Jul 12 06:56:07 mindpipe kernel:  [rt_secret_rebuild+14/48] rt_secret_rebuild+0xe/0x30
Jul 12 06:56:07 mindpipe kernel:  [run_timer_softirq+201/416] run_timer_softirq+0xc9/0x1a0
Jul 12 06:56:07 mindpipe kernel:  [__do_softirq+131/144] __do_softirq+0x83/0x90
Jul 12 06:56:07 mindpipe kernel:  [do_softirq+53/64] do_softirq+0x35/0x40
Jul 12 06:56:07 mindpipe kernel:  [do_IRQ+261/352] do_IRQ+0x105/0x160
Jul 12 06:56:07 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 06:56:07 mindpipe kernel:
Jul 12 07:46:07 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 07:46:07 mindpipe kernel:  [__crc_totalram_pages+1387264/5353478] snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 07:46:07 mindpipe kernel:  [__crc_totalram_pages+1455070/5353478] snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 07:46:07 mindpipe kernel:  [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 07:46:07 mindpipe kernel:  [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 07:46:07 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 07:46:07 mindpipe kernel:  [preempt_schedule+1/80] preempt_schedule+0x1/0x50
Jul 12 07:46:07 mindpipe kernel:  [rt_run_flush+135/144] rt_run_flush+0x87/0x90
Jul 12 07:46:07 mindpipe kernel:  [rt_secret_rebuild+0/48] rt_secret_rebuild+0x0/0x30
Jul 12 07:46:07 mindpipe kernel:  [rt_secret_rebuild+14/48] rt_secret_rebuild+0xe/0x30
Jul 12 07:46:07 mindpipe kernel:  [run_timer_softirq+201/416] run_timer_softirq+0xc9/0x1a0
Jul 12 07:46:07 mindpipe kernel:  [__do_softirq+131/144] __do_softirq+0x83/0x90
Jul 12 07:46:07 mindpipe kernel:  [do_softirq+53/64] do_softirq+0x35/0x40
Jul 12 07:46:07 mindpipe kernel:  [do_IRQ+261/352] do_IRQ+0x105/0x160
Jul 12 07:46:07 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 07:46:07 mindpipe kernel:  [default_idle+35/64] default_idle+0x23/0x40
Jul 12 07:46:07 mindpipe kernel:  [apm_cpu_idle+165/384] apm_cpu_idle+0xa5/0x180
Jul 12 07:46:07 mindpipe kernel:  [cpu_idle+45/64] cpu_idle+0x2d/0x40
Jul 12 07:46:07 mindpipe kernel:  [start_kernel+392/464] start_kernel+0x188/0x1d0
Jul 12 07:46:07 mindpipe kernel:
Jul 12 08:06:07 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 08:06:07 mindpipe kernel:  [__crc_totalram_pages+1387264/5353478] snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 08:06:07 mindpipe kernel:  [__crc_totalram_pages+1455070/5353478] snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 08:06:07 mindpipe kernel:  [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 08:06:07 mindpipe kernel:  [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 08:06:07 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 08:06:07 mindpipe kernel:  [local_bh_enable+46/112] local_bh_enable+0x2e/0x70
Jul 12 08:06:07 mindpipe kernel:  [rt_run_flush+92/144] rt_run_flush+0x5c/0x90
Jul 12 08:06:07 mindpipe kernel:  [rt_secret_rebuild+0/48] rt_secret_rebuild+0x0/0x30
Jul 12 08:06:07 mindpipe kernel:  [rt_secret_rebuild+14/48] rt_secret_rebuild+0xe/0x30
Jul 12 08:06:07 mindpipe kernel:  [run_timer_softirq+201/416] run_timer_softirq+0xc9/0x1a0
Jul 12 08:06:07 mindpipe kernel:  [__do_softirq+131/144] __do_softirq+0x83/0x90
Jul 12 08:06:07 mindpipe kernel:  [do_softirq+53/64] do_softirq+0x35/0x40
Jul 12 08:06:07 mindpipe kernel:  [do_IRQ+261/352] do_IRQ+0x105/0x160
Jul 12 08:06:07 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 08:06:07 mindpipe kernel:
Jul 12 08:26:06 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 08:26:06 mindpipe kernel:  [__crc_totalram_pages+1387264/5353478] snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 08:26:06 mindpipe kernel:  [ide_build_sglist+50/160] ide_build_sglist+0x32/0xa0
Jul 12 08:26:06 mindpipe kernel:  [__crc_totalram_pages+1455070/5353478] snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 08:26:06 mindpipe kernel:  [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 08:26:06 mindpipe kernel:  [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 08:26:06 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 08:26:06 mindpipe kernel:  [del_timer+110/144] del_timer+0x6e/0x90
Jul 12 08:26:06 mindpipe kernel:  [schedule_timeout+94/176] schedule_timeout+0x5e/0xb0
Jul 12 08:26:06 mindpipe kernel:  [process_timeout+0/16] process_timeout+0x0/0x10
Jul 12 08:26:06 mindpipe kernel:  [do_poll+161/192] do_poll+0xa1/0xc0
Jul 12 08:26:06 mindpipe kernel:  [sys_poll+330/560] sys_poll+0x14a/0x230
Jul 12 08:26:06 mindpipe kernel:  [__pollwait+0/160] __pollwait+0x0/0xa0
Jul 12 08:26:06 mindpipe kernel:  [syscall_call+7/11] syscall_call+0x7/0xb
Jul 12 08:26:06 mindpipe kernel:
Jul 12 08:36:06 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 08:36:06 mindpipe kernel:  [__crc_totalram_pages+1387264/5353478] snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 08:36:06 mindpipe kernel:  [__crc_totalram_pages+1455070/5353478] snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 08:36:06 mindpipe kernel:  [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 08:36:06 mindpipe kernel:  [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 08:36:06 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 08:36:06 mindpipe kernel:  [rt_run_flush+66/144] rt_run_flush+0x42/0x90
Jul 12 08:36:06 mindpipe kernel:  [rt_secret_rebuild+0/48] rt_secret_rebuild+0x0/0x30
Jul 12 08:36:06 mindpipe kernel:  [rt_secret_rebuild+14/48] rt_secret_rebuild+0xe/0x30
Jul 12 08:36:06 mindpipe kernel:  [run_timer_softirq+201/416] run_timer_softirq+0xc9/0x1a0
Jul 12 08:36:06 mindpipe kernel:  [__do_softirq+131/144] __do_softirq+0x83/0x90
Jul 12 08:36:06 mindpipe kernel:  [do_softirq+53/64] do_softirq+0x35/0x40
Jul 12 08:36:06 mindpipe kernel:  [do_IRQ+261/352] do_IRQ+0x105/0x160
Jul 12 08:36:06 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 08:36:06 mindpipe kernel:  [default_idle+35/64] default_idle+0x23/0x40
Jul 12 08:36:06 mindpipe kernel:  [apm_cpu_idle+165/384] apm_cpu_idle+0xa5/0x180
Jul 12 08:36:06 mindpipe kernel:  [cpu_idle+45/64] cpu_idle+0x2d/0x40
Jul 12 08:36:06 mindpipe kernel:  [start_kernel+392/464] start_kernel+0x188/0x1d0
Jul 12 08:36:06 mindpipe kernel:
Jul 12 08:46:06 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 08:46:06 mindpipe kernel:  [__crc_totalram_pages+1387264/5353478] snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 08:46:06 mindpipe kernel:  [__crc_totalram_pages+1455070/5353478] snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 08:46:06 mindpipe kernel:  [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 08:46:06 mindpipe kernel:  [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 08:46:06 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 08:46:06 mindpipe kernel:  [rt_run_flush+123/144] rt_run_flush+0x7b/0x90
Jul 12 08:46:06 mindpipe kernel:  [rt_secret_rebuild+0/48] rt_secret_rebuild+0x0/0x30
Jul 12 08:46:06 mindpipe kernel:  [rt_secret_rebuild+14/48] rt_secret_rebuild+0xe/0x30
Jul 12 08:46:06 mindpipe kernel:  [run_timer_softirq+201/416] run_timer_softirq+0xc9/0x1a0
Jul 12 08:46:06 mindpipe kernel:  [__do_softirq+131/144] __do_softirq+0x83/0x90
Jul 12 08:46:06 mindpipe kernel:  [do_softirq+53/64] do_softirq+0x35/0x40
Jul 12 08:46:06 mindpipe kernel:  [do_IRQ+261/352] do_IRQ+0x105/0x160
Jul 12 08:46:06 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 08:46:06 mindpipe kernel:  [default_idle+35/64] default_idle+0x23/0x40
Jul 12 08:46:06 mindpipe kernel:  [apm_cpu_idle+165/384] apm_cpu_idle+0xa5/0x180
Jul 12 08:46:06 mindpipe kernel:  [cpu_idle+45/64] cpu_idle+0x2d/0x40
Jul 12 08:46:06 mindpipe kernel:  [start_kernel+392/464] start_kernel+0x188/0x1d0
Jul 12 08:46:06 mindpipe kernel:
Jul 12 08:56:06 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 08:56:06 mindpipe kernel:  [__crc_totalram_pages+1387264/5353478] snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 08:56:06 mindpipe kernel:  [__crc_totalram_pages+1455070/5353478] snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 08:56:06 mindpipe kernel:  [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 08:56:06 mindpipe kernel:  [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 08:56:06 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 08:56:06 mindpipe kernel:  [rt_run_flush+123/144] rt_run_flush+0x7b/0x90
Jul 12 08:56:06 mindpipe kernel:  [rt_secret_rebuild+0/48] rt_secret_rebuild+0x0/0x30
Jul 12 08:56:06 mindpipe kernel:  [rt_secret_rebuild+14/48] rt_secret_rebuild+0xe/0x30
Jul 12 08:56:06 mindpipe kernel:  [run_timer_softirq+201/416] run_timer_softirq+0xc9/0x1a0
Jul 12 08:56:06 mindpipe kernel:  [__do_softirq+131/144] __do_softirq+0x83/0x90
Jul 12 08:56:06 mindpipe kernel:  [do_softirq+53/64] do_softirq+0x35/0x40
Jul 12 08:56:06 mindpipe kernel:  [do_IRQ+261/352] do_IRQ+0x105/0x160
Jul 12 08:56:06 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 08:56:06 mindpipe kernel:  [default_idle+35/64] default_idle+0x23/0x40
Jul 12 08:56:06 mindpipe kernel:  [apm_cpu_idle+165/384] apm_cpu_idle+0xa5/0x180
Jul 12 08:56:06 mindpipe kernel:  [cpu_idle+45/64] cpu_idle+0x2d/0x40
Jul 12 08:56:06 mindpipe kernel:  [start_kernel+392/464] start_kernel+0x188/0x1d0
Jul 12 08:56:06 mindpipe kernel:
Jul 12 09:06:06 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 09:06:06 mindpipe kernel:  [__crc_totalram_pages+1387264/5353478] snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 09:06:06 mindpipe kernel:  [__crc_totalram_pages+1455070/5353478] snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 09:06:06 mindpipe kernel:  [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 09:06:06 mindpipe kernel:  [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 09:06:06 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 09:06:06 mindpipe kernel:  [preempt_schedule+1/80] preempt_schedule+0x1/0x50
Jul 12 09:06:06 mindpipe kernel:  [rt_run_flush+135/144] rt_run_flush+0x87/0x90
Jul 12 09:06:06 mindpipe kernel:  [rt_secret_rebuild+0/48] rt_secret_rebuild+0x0/0x30
Jul 12 09:06:06 mindpipe kernel:  [rt_secret_rebuild+14/48] rt_secret_rebuild+0xe/0x30
Jul 12 09:06:06 mindpipe kernel:  [run_timer_softirq+201/416] run_timer_softirq+0xc9/0x1a0
Jul 12 09:06:06 mindpipe kernel:  [__do_softirq+131/144] __do_softirq+0x83/0x90
Jul 12 09:06:06 mindpipe kernel:  [do_softirq+53/64] do_softirq+0x35/0x40
Jul 12 09:06:06 mindpipe kernel:  [do_IRQ+261/352] do_IRQ+0x105/0x160
Jul 12 09:06:06 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 09:06:06 mindpipe kernel:  [default_idle+35/64] default_idle+0x23/0x40
Jul 12 09:06:06 mindpipe kernel:  [apm_cpu_idle+165/384] apm_cpu_idle+0xa5/0x180
Jul 12 09:06:06 mindpipe kernel:  [cpu_idle+45/64] cpu_idle+0x2d/0x40
Jul 12 09:06:06 mindpipe kernel:  [start_kernel+392/464] start_kernel+0x188/0x1d0
Jul 12 09:06:06 mindpipe kernel:
Jul 12 09:36:06 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 09:36:06 mindpipe kernel:  [__crc_totalram_pages+1387264/5353478] snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 09:36:06 mindpipe kernel:  [__crc_totalram_pages+1455070/5353478] snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 09:36:06 mindpipe kernel:  [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 09:36:06 mindpipe kernel:  [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 09:36:06 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 09:36:06 mindpipe kernel:  [local_bh_enable+23/112] local_bh_enable+0x17/0x70
Jul 12 09:36:06 mindpipe kernel:  [rt_run_flush+92/144] rt_run_flush+0x5c/0x90
Jul 12 09:36:06 mindpipe kernel:  [rt_secret_rebuild+0/48] rt_secret_rebuild+0x0/0x30
Jul 12 09:36:06 mindpipe kernel:  [rt_secret_rebuild+14/48] rt_secret_rebuild+0xe/0x30
Jul 12 09:36:06 mindpipe kernel:  [run_timer_softirq+201/416] run_timer_softirq+0xc9/0x1a0
Jul 12 09:36:06 mindpipe kernel:  [__do_softirq+131/144] __do_softirq+0x83/0x90
Jul 12 09:36:06 mindpipe kernel:  [do_softirq+53/64] do_softirq+0x35/0x40
Jul 12 09:36:06 mindpipe kernel:  [do_IRQ+261/352] do_IRQ+0x105/0x160
Jul 12 09:36:06 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 09:36:06 mindpipe kernel:  [default_idle+35/64] default_idle+0x23/0x40
Jul 12 09:36:06 mindpipe kernel:  [apm_cpu_idle+165/384] apm_cpu_idle+0xa5/0x180
Jul 12 09:36:06 mindpipe kernel:  [cpu_idle+45/64] cpu_idle+0x2d/0x40
Jul 12 09:36:06 mindpipe kernel:  [start_kernel+392/464] start_kernel+0x188/0x1d0
Jul 12 09:36:06 mindpipe kernel:
Jul 12 09:46:06 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 09:46:06 mindpipe kernel:  [__crc_totalram_pages+1387264/5353478] snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 09:46:06 mindpipe kernel:  [__crc_totalram_pages+1455070/5353478] snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 09:46:06 mindpipe kernel:  [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 09:46:06 mindpipe kernel:  [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 09:46:06 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 09:46:06 mindpipe kernel:  [preempt_schedule+23/80] preempt_schedule+0x17/0x50
Jul 12 09:46:06 mindpipe kernel:  [rt_run_flush+92/144] rt_run_flush+0x5c/0x90
Jul 12 09:46:06 mindpipe kernel:  [rt_secret_rebuild+0/48] rt_secret_rebuild+0x0/0x30
Jul 12 09:46:06 mindpipe kernel:  [rt_secret_rebuild+14/48] rt_secret_rebuild+0xe/0x30
Jul 12 09:46:06 mindpipe kernel:  [run_timer_softirq+201/416] run_timer_softirq+0xc9/0x1a0
Jul 12 09:46:06 mindpipe kernel:  [__do_softirq+131/144] __do_softirq+0x83/0x90
Jul 12 09:46:06 mindpipe kernel:  [do_softirq+53/64] do_softirq+0x35/0x40
Jul 12 09:46:06 mindpipe kernel:  [do_IRQ+261/352] do_IRQ+0x105/0x160
Jul 12 09:46:06 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 09:46:06 mindpipe kernel:  [default_idle+35/64] default_idle+0x23/0x40
Jul 12 09:46:06 mindpipe kernel:  [apm_cpu_idle+165/384] apm_cpu_idle+0xa5/0x180
Jul 12 09:46:06 mindpipe kernel:  [cpu_idle+45/64] cpu_idle+0x2d/0x40
Jul 12 09:46:06 mindpipe kernel:  [start_kernel+392/464] start_kernel+0x188/0x1d0
Jul 12 09:46:06 mindpipe kernel:
Jul 12 10:16:05 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 10:16:05 mindpipe kernel:  [__crc_totalram_pages+1387264/5353478] snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 10:16:05 mindpipe kernel:  [__crc_totalram_pages+1455070/5353478] snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 10:16:05 mindpipe kernel:  [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 10:16:05 mindpipe kernel:  [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 10:16:05 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 10:16:05 mindpipe kernel:  [preempt_schedule+73/80] preempt_schedule+0x49/0x50
Jul 12 10:16:05 mindpipe kernel:  [rt_run_flush+135/144] rt_run_flush+0x87/0x90
Jul 12 10:16:05 mindpipe kernel:  [rt_secret_rebuild+0/48] rt_secret_rebuild+0x0/0x30
Jul 12 10:16:05 mindpipe kernel:  [rt_secret_rebuild+14/48] rt_secret_rebuild+0xe/0x30
Jul 12 10:16:05 mindpipe kernel:  [run_timer_softirq+201/416] run_timer_softirq+0xc9/0x1a0
Jul 12 10:16:05 mindpipe kernel:  [__do_softirq+131/144] __do_softirq+0x83/0x90
Jul 12 10:16:05 mindpipe kernel:  [do_softirq+53/64] do_softirq+0x35/0x40
Jul 12 10:16:05 mindpipe kernel:  [do_IRQ+261/352] do_IRQ+0x105/0x160
Jul 12 10:16:05 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 10:16:05 mindpipe kernel:  [default_idle+35/64] default_idle+0x23/0x40
Jul 12 10:16:05 mindpipe kernel:  [apm_cpu_idle+165/384] apm_cpu_idle+0xa5/0x180
Jul 12 10:16:05 mindpipe kernel:  [cpu_idle+45/64] cpu_idle+0x2d/0x40
Jul 12 10:16:05 mindpipe kernel:  [start_kernel+392/464] start_kernel+0x188/0x1d0
Jul 12 10:16:05 mindpipe kernel:
Jul 12 10:46:05 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 10:46:05 mindpipe kernel:  [__crc_totalram_pages+1387264/5353478] snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 10:46:05 mindpipe kernel:  [__crc_totalram_pages+1455070/5353478] snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 10:46:05 mindpipe kernel:  [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 10:46:05 mindpipe kernel:  [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 10:46:05 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 10:46:05 mindpipe kernel:  [rt_run_flush+94/144] rt_run_flush+0x5e/0x90
Jul 12 10:46:05 mindpipe kernel:  [rt_secret_rebuild+0/48] rt_secret_rebuild+0x0/0x30
Jul 12 10:46:05 mindpipe kernel:  [rt_secret_rebuild+14/48] rt_secret_rebuild+0xe/0x30
Jul 12 10:46:05 mindpipe kernel:  [run_timer_softirq+201/416] run_timer_softirq+0xc9/0x1a0
Jul 12 10:46:05 mindpipe kernel:  [__do_softirq+131/144] __do_softirq+0x83/0x90
Jul 12 10:46:05 mindpipe kernel:  [do_softirq+53/64] do_softirq+0x35/0x40
Jul 12 10:46:05 mindpipe kernel:  [do_IRQ+261/352] do_IRQ+0x105/0x160
Jul 12 10:46:05 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 10:46:05 mindpipe kernel:  [default_idle+35/64] default_idle+0x23/0x40
Jul 12 10:46:05 mindpipe kernel:  [apm_cpu_idle+165/384] apm_cpu_idle+0xa5/0x180
Jul 12 10:46:05 mindpipe kernel:  [cpu_idle+45/64] cpu_idle+0x2d/0x40
Jul 12 10:46:05 mindpipe kernel:  [start_kernel+392/464] start_kernel+0x188/0x1d0
Jul 12 10:46:05 mindpipe kernel:
Jul 12 10:56:05 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 10:56:05 mindpipe kernel:  [__crc_totalram_pages+1387264/5353478] snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 10:56:05 mindpipe kernel:  [__crc_totalram_pages+1455070/5353478] snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 10:56:05 mindpipe kernel:  [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 10:56:05 mindpipe kernel:  [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 10:56:05 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 10:56:05 mindpipe kernel:  [local_bh_enable+8/112] local_bh_enable+0x8/0x70
Jul 12 10:56:05 mindpipe kernel:  [rt_run_flush+92/144] rt_run_flush+0x5c/0x90
Jul 12 10:56:05 mindpipe kernel:  [rt_secret_rebuild+0/48] rt_secret_rebuild+0x0/0x30
Jul 12 10:56:05 mindpipe kernel:  [rt_secret_rebuild+14/48] rt_secret_rebuild+0xe/0x30
Jul 12 10:56:05 mindpipe kernel:  [run_timer_softirq+201/416] run_timer_softirq+0xc9/0x1a0
Jul 12 10:56:05 mindpipe kernel:  [__do_softirq+131/144] __do_softirq+0x83/0x90
Jul 12 10:56:05 mindpipe kernel:  [do_softirq+53/64] do_softirq+0x35/0x40
Jul 12 10:56:05 mindpipe kernel:  [do_IRQ+261/352] do_IRQ+0x105/0x160
Jul 12 10:56:05 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 10:56:05 mindpipe kernel:  [default_idle+35/64] default_idle+0x23/0x40
Jul 12 10:56:05 mindpipe kernel:  [apm_cpu_idle+165/384] apm_cpu_idle+0xa5/0x180
Jul 12 10:56:05 mindpipe kernel:  [cpu_idle+45/64] cpu_idle+0x2d/0x40
Jul 12 10:56:05 mindpipe kernel:  [start_kernel+392/464] start_kernel+0x188/0x1d0
Jul 12 10:56:05 mindpipe kernel:
Jul 12 11:06:05 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 11:06:05 mindpipe kernel:  [__crc_totalram_pages+1387264/5353478] snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 11:06:05 mindpipe kernel:  [__crc_totalram_pages+1455070/5353478] snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 11:06:05 mindpipe kernel:  [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 11:06:05 mindpipe kernel:  [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 11:06:05 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 11:06:05 mindpipe kernel:  [preempt_schedule+9/80] preempt_schedule+0x9/0x50
Jul 12 11:06:05 mindpipe kernel:  [rt_run_flush+92/144] rt_run_flush+0x5c/0x90
Jul 12 11:06:05 mindpipe kernel:  [rt_secret_rebuild+0/48] rt_secret_rebuild+0x0/0x30
Jul 12 11:06:05 mindpipe kernel:  [rt_secret_rebuild+14/48] rt_secret_rebuild+0xe/0x30
Jul 12 11:06:05 mindpipe kernel:  [run_timer_softirq+201/416] run_timer_softirq+0xc9/0x1a0
Jul 12 11:06:05 mindpipe kernel:  [__do_softirq+131/144] __do_softirq+0x83/0x90
Jul 12 11:06:05 mindpipe kernel:  [do_softirq+53/64] do_softirq+0x35/0x40
Jul 12 11:06:05 mindpipe kernel:  [do_IRQ+261/352] do_IRQ+0x105/0x160
Jul 12 11:06:05 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 11:06:05 mindpipe kernel:  [default_idle+35/64] default_idle+0x23/0x40
Jul 12 11:06:05 mindpipe kernel:  [apm_cpu_idle+165/384] apm_cpu_idle+0xa5/0x180
Jul 12 11:06:05 mindpipe kernel:  [cpu_idle+45/64] cpu_idle+0x2d/0x40
Jul 12 11:06:05 mindpipe kernel:  [start_kernel+392/464] start_kernel+0x188/0x1d0
Jul 12 11:06:05 mindpipe kernel:
Jul 12 11:36:05 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 11:36:05 mindpipe kernel:  [__crc_totalram_pages+1387264/5353478] snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 11:36:05 mindpipe kernel:  [__crc_totalram_pages+1455070/5353478] snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 11:36:05 mindpipe kernel:  [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 11:36:05 mindpipe kernel:  [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 11:36:05 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 11:36:05 mindpipe kernel:  [local_bh_enable+0/112] local_bh_enable+0x0/0x70
Jul 12 11:36:05 mindpipe kernel:  [rt_run_flush+92/144] rt_run_flush+0x5c/0x90
Jul 12 11:36:05 mindpipe kernel:  [rt_secret_rebuild+0/48] rt_secret_rebuild+0x0/0x30
Jul 12 11:36:05 mindpipe kernel:  [rt_secret_rebuild+14/48] rt_secret_rebuild+0xe/0x30
Jul 12 11:36:05 mindpipe kernel:  [run_timer_softirq+201/416] run_timer_softirq+0xc9/0x1a0
Jul 12 11:36:05 mindpipe kernel:  [__do_softirq+131/144] __do_softirq+0x83/0x90
Jul 12 11:36:05 mindpipe kernel:  [do_softirq+53/64] do_softirq+0x35/0x40
Jul 12 11:36:05 mindpipe kernel:  [do_IRQ+261/352] do_IRQ+0x105/0x160
Jul 12 11:36:05 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 11:36:05 mindpipe kernel:  [default_idle+35/64] default_idle+0x23/0x40
Jul 12 11:36:05 mindpipe kernel:  [apm_cpu_idle+165/384] apm_cpu_idle+0xa5/0x180
Jul 12 11:36:05 mindpipe kernel:  [cpu_idle+45/64] cpu_idle+0x2d/0x40
Jul 12 11:36:05 mindpipe kernel:  [start_kernel+392/464] start_kernel+0x188/0x1d0
Jul 12 11:36:05 mindpipe kernel:
Jul 12 11:46:05 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 11:46:05 mindpipe kernel:  [__crc_totalram_pages+1387264/5353478] snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 11:46:05 mindpipe kernel:  [__crc_totalram_pages+1455070/5353478] snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 11:46:05 mindpipe kernel:  [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 11:46:05 mindpipe kernel:  [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 11:46:05 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 11:46:05 mindpipe kernel:  [rt_run_flush+66/144] rt_run_flush+0x42/0x90
Jul 12 11:46:05 mindpipe kernel:  [rt_secret_rebuild+0/48] rt_secret_rebuild+0x0/0x30
Jul 12 11:46:05 mindpipe kernel:  [rt_secret_rebuild+14/48] rt_secret_rebuild+0xe/0x30
Jul 12 11:46:05 mindpipe kernel:  [run_timer_softirq+201/416] run_timer_softirq+0xc9/0x1a0
Jul 12 11:46:05 mindpipe kernel:  [__do_softirq+131/144] __do_softirq+0x83/0x90
Jul 12 11:46:05 mindpipe kernel:  [do_softirq+53/64] do_softirq+0x35/0x40
Jul 12 11:46:05 mindpipe kernel:  [do_IRQ+261/352] do_IRQ+0x105/0x160
Jul 12 11:46:05 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 11:46:05 mindpipe kernel:  [default_idle+35/64] default_idle+0x23/0x40
Jul 12 11:46:05 mindpipe kernel:  [apm_cpu_idle+165/384] apm_cpu_idle+0xa5/0x180
Jul 12 11:46:05 mindpipe kernel:  [cpu_idle+45/64] cpu_idle+0x2d/0x40
Jul 12 11:46:05 mindpipe kernel:  [start_kernel+392/464] start_kernel+0x188/0x1d0

That continues until I sat down at the machine and switched consoles.  I am aware that 
this is a known latency 'hot spot' and not trivial to fix.  Here are the traces anyway:

Jul 12 17:33:42 mindpipe kernel:
Jul 12 17:33:42 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p Jul 12 17:33:42 mindpipe kernel:  [__crc_totalram_pages+1387264/5353478] snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 17:33:42 mindpipe kernel:  [__crc_totalram_pages+1455070/5353478] snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 17:33:42 mindpipe kernel:  [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 17:33:42 mindpipe kernel:  [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 17:33:42 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 17:33:42 mindpipe kernel:  [release_console_sem+129/208] release_console_sem+0x81/0xd0
Jul 12 17:33:42 mindpipe kernel:  [do_con_write+657/1888] do_con_write+0x291/0x760
Jul 12 17:33:42 mindpipe kernel:  [con_put_char+51/64] con_put_char+0x33/0x40
Jul 12 17:33:42 mindpipe kernel:  [opost+162/464] opost+0xa2/0x1d0
Jul 12 17:33:42 mindpipe kernel:  [write_chan+437/544] write_chan+0x1b5/0x220
Jul 12 17:33:42 mindpipe kernel:  [default_wake_function+0/32] default_wake_function+0x0/0x20
Jul 12 17:33:42 mindpipe kernel:  [default_wake_function+0/32] default_wake_function+0x0/0x20
Jul 12 17:33:42 mindpipe kernel:  [tty_write+434/608] tty_write+0x1b2/0x260
Jul 12 17:33:42 mindpipe kernel:  [write_chan+0/544] write_chan+0x0/0x220
Jul 12 17:33:42 mindpipe kernel:  [vfs_write+186/256] vfs_write+0xba/0x100
Jul 12 17:33:42 mindpipe kernel:  [sys_write+45/80] sys_write+0x2d/0x50
Jul 12 17:33:42 mindpipe kernel:  [syscall_call+7/11] syscall_call+0x7/0xb
Jul 12 17:33:42 mindpipe kernel:
Jul 12 17:33:42 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 17:33:42 mindpipe kernel:  [__crc_totalram_pages+1387264/5353478] snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 17:33:42 mindpipe kernel:  [avc_has_perm+72/96] avc_has_perm+0x48/0x60
Jul 12 17:33:42 mindpipe kernel:  [__crc_totalram_pages+1455070/5353478] snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 17:33:42 mindpipe kernel:  [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 17:33:42 mindpipe kernel:  [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 17:33:42 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 17:33:42 mindpipe kernel:  [lf+1/112] lf+0x1/0x70
Jul 12 17:33:42 mindpipe kernel:  [do_con_trol+2907/3360] do_con_trol+0xb5b/0xd20
Jul 12 17:33:42 mindpipe kernel:  [do_con_write+1128/1888] do_con_write+0x468/0x760
Jul 12 17:33:42 mindpipe kernel:  [con_put_char+51/64] con_put_char+0x33/0x40
Jul 12 17:33:42 mindpipe kernel:  [opost+162/464] opost+0xa2/0x1d0
Jul 12 17:33:42 mindpipe kernel:  [write_chan+437/544] write_chan+0x1b5/0x220
Jul 12 17:33:42 mindpipe kernel:  [default_wake_function+0/32] default_wake_function+0x0/0x20
Jul 12 17:33:42 mindpipe kernel:  [default_wake_function+0/32] default_wake_function+0x0/0x20
Jul 12 17:33:42 mindpipe kernel:  [tty_write+434/608] tty_write+0x1b2/0x260
Jul 12 17:33:42 mindpipe kernel:  [write_chan+0/544] write_chan+0x0/0x220
Jul 12 17:33:42 mindpipe kernel:  [vfs_write+186/256] vfs_write+0xba/0x100
Jul 12 17:33:42 mindpipe kernel:  [sys_write+45/80] sys_write+0x2d/0x50
Jul 12 17:33:42 mindpipe kernel:  [syscall_call+7/11] syscall_call+0x7/0xb
Jul 12 17:33:42 mindpipe kernel:

It looks like reiserfs has a hot spot as well:

Jul 12 17:55:11 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 17:55:11 mindpipe kernel:  [__crc_totalram_pages+1387264/5353478] snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 17:55:11 mindpipe kernel:  [do_pollfd+125/144] do_pollfd+0x7d/0x90
Jul 12 17:55:11 mindpipe kernel:  [__crc_totalram_pages+1455070/5353478] snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 17:55:11 mindpipe kernel:  [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 17:55:11 mindpipe kernel:  [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 17:55:11 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 17:55:11 mindpipe kernel:
Jul 12 17:55:11 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 17:55:11 mindpipe kernel:  [__crc_totalram_pages+1387264/5353478] snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 17:55:11 mindpipe kernel:  [__crc_totalram_pages+1455070/5353478] snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 17:55:11 mindpipe kernel:  [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 17:55:11 mindpipe kernel:  [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 17:55:11 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 17:55:11 mindpipe kernel:  [free_hot_cold_page+21/288] free_hot_cold_page+0x15/0x120
Jul 12 17:55:11 mindpipe kernel:  [poll_freewait+68/80] poll_freewait+0x44/0x50
Jul 12 17:55:11 mindpipe kernel:  [sys_poll+489/560] sys_poll+0x1e9/0x230
Jul 12 17:55:11 mindpipe kernel:  [__pollwait+0/160] __pollwait+0x0/0xa0
Jul 12 17:55:11 mindpipe kernel:  [syscall_call+7/11] syscall_call+0x7/0xb
Jul 12 17:55:11 mindpipe kernel:
Jul 12 17:55:11 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 17:55:11 mindpipe kernel:  [__crc_totalram_pages+1387264/5353478] snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 17:55:11 mindpipe kernel:  [__crc_totalram_pages+1455070/5353478] snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 17:55:11 mindpipe kernel:  [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 17:55:11 mindpipe kernel:  [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 17:55:11 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 17:55:11 mindpipe kernel:  [can_dirty+99/192] can_dirty+0x63/0xc0
Jul 12 17:55:11 mindpipe kernel:  [dirty_one_transaction+74/128] dirty_one_transaction+0x4a/0x80
Jul 12 17:55:11 mindpipe kernel:  [flush_commit_list+519/944] flush_commit_list+0x207/0x3b0
Jul 12 17:55:11 mindpipe kernel:  [do_journal_end+2081/2848] do_journal_end+0x821/0xb20
Jul 12 17:55:11 mindpipe kernel:  [journal_end_sync+54/112] journal_end_sync+0x36/0x70
Jul 12 17:55:11 mindpipe kernel:  [__commit_trans_jl+223/240] __commit_trans_jl+0xdf/0xf0
Jul 12 17:55:11 mindpipe kernel:  [reiserfs_commit_for_inode+32/64] reiserfs_commit_for_inode+0x20/0x40
Jul 12 17:55:11 mindpipe kernel:  [reiserfs_sync_file+77/144] reiserfs_sync_file+0x4d/0x90
Jul 12 17:55:11 mindpipe kernel:  [sys_fsync+123/176] sys_fsync+0x7b/0xb0
Jul 12 17:55:11 mindpipe kernel:  [syscall_call+7/11] syscall_call+0x7/0xb
Jul 12 17:55:11 mindpipe kernel:
Jul 12 17:55:11 mindpipe kernel:
Jul 12 17:55:11 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 17:55:11 mindpipe kernel:  [__crc_totalram_pages+1387264/5353478] snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 17:55:11 mindpipe kernel:  [__crc_totalram_pages+1455070/5353478] snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 17:55:11 mindpipe kernel:  [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 17:55:11 mindpipe kernel:  [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 17:55:11 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 17:55:11 mindpipe kernel:  [finish_task_switch+34/112] finish_task_switch+0x22/0x70
Jul 12 17:55:11 mindpipe kernel:  [schedule+731/1424] schedule+0x2db/0x590
Jul 12 17:55:11 mindpipe kernel:  [__mod_timer+206/384] __mod_timer+0xce/0x180
Jul 12 17:55:11 mindpipe kernel:  [schedule_timeout+88/176] schedule_timeout+0x58/0xb0
Jul 12 17:55:11 mindpipe kernel:  [process_timeout+0/16] process_timeout+0x0/0x10
Jul 12 17:55:11 mindpipe kernel:  [do_poll+161/192] do_poll+0xa1/0xc0
Jul 12 17:55:11 mindpipe kernel:  [sys_poll+330/560] sys_poll+0x14a/0x230
Jul 12 17:55:11 mindpipe kernel:  [__pollwait+0/160] __pollwait+0x0/0xa0
Jul 12 17:55:11 mindpipe kernel:  [syscall_call+7/11] syscall_call+0x7/0xb
Jul 12 17:55:11 mindpipe kernel:
Jul 12 17:55:17 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 17:55:17 mindpipe kernel:  [__crc_totalram_pages+1387264/5353478] snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 17:55:17 mindpipe kernel:  [__crc_totalram_pages+1455070/5353478] snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 17:55:17 mindpipe kernel:  [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 17:55:17 mindpipe kernel:  [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 17:55:17 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 17:55:17 mindpipe kernel:  [remove_journal_hash+72/208] remove_journal_hash+0x48/0xd0
Jul 12 17:55:17 mindpipe kernel:  [remove_all_from_journal_list+78/160] remove_all_from_journal_list+0x4e/0xa0
Jul 12 17:55:17 mindpipe kernel:  [flush_journal_list+496/1344] flush_journal_list+0x1f0/0x540
Jul 12 17:55:17 mindpipe kernel:  [flush_older_journal_lists+48/64] flush_older_journal_lists+0x30/0x40
Jul 12 17:55:17 mindpipe kernel:  [flush_journal_list+795/1344] flush_journal_list+0x31b/0x540
Jul 12 17:55:17 mindpipe kernel:  [flush_used_journal_lists+147/160] flush_used_journal_lists+0x93/0xa0
Jul 12 17:55:17 mindpipe kernel:  [flush_old_journal_lists+69/112] flush_old_journal_lists+0x45/0x70
Jul 12 17:55:17 mindpipe kernel:  [do_journal_end+1853/2848] do_journal_end+0x73d/0xb20
Jul 12 17:55:17 mindpipe kernel:  [journal_end_sync+54/112] journal_end_sync+0x36/0x70
Jul 12 17:55:17 mindpipe kernel:  [__commit_trans_jl+223/240] __commit_trans_jl+0xdf/0xf0
Jul 12 17:55:17 mindpipe kernel:  [reiserfs_commit_for_inode+32/64] reiserfs_commit_for_inode+0x20/0x40
Jul 12 17:55:17 mindpipe kernel:  [reiserfs_sync_file+77/144] reiserfs_sync_file+0x4d/0x90
Jul 12 17:55:17 mindpipe kernel:  [sys_fsync+123/176] sys_fsync+0x7b/0xb0
Jul 12 17:55:17 mindpipe kernel:  [syscall_call+7/11] syscall_call+0x7/0xb
Jul 12 17:55:17 mindpipe kernel:
Jul 12 17:55:17 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 17:55:17 mindpipe kernel:  [__crc_totalram_pages+1387264/5353478] snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 17:55:17 mindpipe kernel:  [__crc_totalram_pages+1455070/5353478] snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 17:55:17 mindpipe kernel:  [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 17:55:17 mindpipe kernel:  [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 17:55:17 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 17:55:17 mindpipe kernel:  [remove_journal_hash+67/208] remove_journal_hash+0x43/0xd0
Jul 12 17:55:17 mindpipe kernel:  [remove_all_from_journal_list+78/160] remove_all_from_journal_list+0x4e/0xa0
Jul 12 17:55:17 mindpipe kernel:  [flush_journal_list+496/1344] flush_journal_list+0x1f0/0x540
Jul 12 17:55:17 mindpipe kernel:  [flush_older_journal_lists+48/64] flush_older_journal_lists+0x30/0x40
Jul 12 17:55:17 mindpipe kernel:  [flush_journal_list+795/1344] flush_journal_list+0x31b/0x540
Jul 12 17:55:17 mindpipe kernel:  [flush_used_journal_lists+147/160] flush_used_journal_lists+0x93/0xa0
Jul 12 17:55:17 mindpipe kernel:  [flush_old_journal_lists+69/112] flush_old_journal_lists+0x45/0x70
Jul 12 17:55:17 mindpipe kernel:  [do_journal_end+1853/2848] do_journal_end+0x73d/0xb20
Jul 12 17:55:17 mindpipe kernel:  [journal_end_sync+54/112] journal_end_sync+0x36/0x70
Jul 12 17:55:17 mindpipe kernel:  [__commit_trans_jl+223/240] __commit_trans_jl+0xdf/0xf0
Jul 12 17:55:17 mindpipe kernel:  [reiserfs_commit_for_inode+32/64] reiserfs_commit_for_inode+0x20/0x40
Jul 12 17:55:17 mindpipe kernel:  [reiserfs_sync_file+77/144] reiserfs_sync_file+0x4d/0x90
Jul 12 17:55:17 mindpipe kernel:  [sys_fsync+123/176] sys_fsync+0x7b/0xb0
Jul 12 17:55:17 mindpipe kernel:  [syscall_call+7/11] syscall_call+0x7/0xb
Jul 12 17:55:17 mindpipe kernel:

A few minutes later:

Jul 12 17:56:54 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 17:56:54 mindpipe kernel:  [__crc_totalram_pages+1387264/5353478] snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 17:56:54 mindpipe kernel:  [i8042_timer_func+0/16] i8042_timer_func+0x0/0x10
Jul 12 17:56:54 mindpipe kernel:  [__crc_totalram_pages+1455070/5353478] snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 17:56:54 mindpipe kernel:  [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 17:56:54 mindpipe kernel:  [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 17:56:54 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 17:56:54 mindpipe kernel:  [can_dirty+144/192] can_dirty+0x90/0xc0
Jul 12 17:56:54 mindpipe kernel:  [dirty_one_transaction+74/128] dirty_one_transaction+0x4a/0x80
Jul 12 17:56:54 mindpipe kernel:  [flush_commit_list+519/944] flush_commit_list+0x207/0x3b0
Jul 12 17:56:54 mindpipe kernel:  [do_journal_end+2081/2848] do_journal_end+0x821/0xb20
Jul 12 17:56:54 mindpipe kernel:  [journal_end_sync+54/112] journal_end_sync+0x36/0x70
Jul 12 17:56:54 mindpipe kernel:  [reiserfs_sync_fs+61/128] reiserfs_sync_fs+0x3d/0x80
Jul 12 17:56:54 mindpipe kernel:  [sync_supers+174/192] sync_supers+0xae/0xc0
Jul 12 17:56:54 mindpipe kernel:  [wb_kupdate+47/256] wb_kupdate+0x2f/0x100
Jul 12 17:56:54 mindpipe kernel:  [deactivate_task+29/64] deactivate_task+0x1d/0x40
Jul 12 17:56:54 mindpipe kernel:  [__pdflush+200/496] __pdflush+0xc8/0x1f0
Jul 12 17:56:54 mindpipe kernel:  [pdflush+0/48] pdflush+0x0/0x30
Jul 12 17:56:54 mindpipe kernel:  [pdflush+30/48] pdflush+0x1e/0x30
Jul 12 17:56:54 mindpipe kernel:  [wb_kupdate+0/256] wb_kupdate+0x0/0x100
Jul 12 17:56:54 mindpipe kernel:  [kthread+163/176] kthread+0xa3/0xb0
Jul 12 17:56:54 mindpipe kernel:  [kthread+0/176] kthread+0x0/0xb0
Jul 12 17:56:54 mindpipe kernel:  [kernel_thread_helper+5/16] kernel_thread_helper+0x5/0x10
Jul 12 17:56:54 mindpipe kernel:

Another one:

Jul 12 17:57:22 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 17:57:22 mindpipe kernel:  [__crc_totalram_pages+1387264/5353478] snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 17:57:22 mindpipe kernel:  [__crc_totalram_pages+1455070/5353478] snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 17:57:22 mindpipe kernel:  [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 17:57:22 mindpipe kernel:  [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 17:57:22 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 17:57:22 mindpipe kernel:  [remove_journal_hash+67/208] remove_journal_hash+0x43/0xd0
Jul 12 17:57:22 mindpipe kernel:  [remove_all_from_journal_list+78/160] remove_all_from_journal_list+0x4e/0xa0
Jul 12 17:57:22 mindpipe kernel:  [flush_journal_list+496/1344] flush_journal_list+0x1f0/0x540
Jul 12 17:57:22 mindpipe kernel:  [flush_older_journal_lists+48/64] flush_older_journal_lists+0x30/0x40
Jul 12 17:57:22 mindpipe kernel:  [flush_journal_list+795/1344] flush_journal_list+0x31b/0x540
Jul 12 17:57:22 mindpipe kernel:  [flush_used_journal_lists+147/160] flush_used_journal_lists+0x93/0xa0
Jul 12 17:57:22 mindpipe kernel:  [flush_old_journal_lists+69/112] flush_old_journal_lists+0x45/0x70
Jul 12 17:57:22 mindpipe kernel:  [do_journal_end+1853/2848] do_journal_end+0x73d/0xb20
Jul 12 17:57:22 mindpipe kernel:  [journal_end_sync+54/112] journal_end_sync+0x36/0x70
Jul 12 17:57:22 mindpipe kernel:  [__commit_trans_jl+223/240] __commit_trans_jl+0xdf/0xf0
Jul 12 17:57:22 mindpipe kernel:  [reiserfs_commit_for_inode+32/64] reiserfs_commit_for_inode+0x20/0x40
Jul 12 17:57:22 mindpipe kernel:  [reiserfs_sync_file+77/144] reiserfs_sync_file+0x4d/0x90
Jul 12 17:57:22 mindpipe kernel:  [sys_fsync+123/176] sys_fsync+0x7b/0xb0
Jul 12 17:57:22 mindpipe kernel:  [syscall_call+7/11] syscall_call+0x7/0xb
Jul 12 17:57:22 mindpipe kernel:
Jul 12 17:57:22 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 17:57:22 mindpipe kernel:  [__crc_totalram_pages+1387264/5353478] snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 17:57:22 mindpipe kernel:  [__crc_totalram_pages+1455070/5353478] snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 17:57:22 mindpipe kernel:  [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 17:57:22 mindpipe kernel:  [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 17:57:22 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 17:57:22 mindpipe kernel:  [remove_journal_hash+72/208] remove_journal_hash+0x48/0xd0
Jul 12 17:57:22 mindpipe kernel:  [remove_all_from_journal_list+78/160] remove_all_from_journal_list+0x4e/0xa0
Jul 12 17:57:22 mindpipe kernel:  [flush_journal_list+496/1344] flush_journal_list+0x1f0/0x540
Jul 12 17:57:22 mindpipe kernel:  [flush_older_journal_lists+48/64] flush_older_journal_lists+0x30/0x40
Jul 12 17:57:22 mindpipe kernel:  [flush_journal_list+795/1344] flush_journal_list+0x31b/0x540
Jul 12 17:57:22 mindpipe kernel:  [flush_used_journal_lists+147/160] flush_used_journal_lists+0x93/0xa0
Jul 12 17:57:22 mindpipe kernel:  [flush_old_journal_lists+69/112] flush_old_journal_lists+0x45/0x70
Jul 12 17:57:22 mindpipe kernel:  [do_journal_end+1853/2848] do_journal_end+0x73d/0xb20
Jul 12 17:57:22 mindpipe kernel:  [journal_end_sync+54/112] journal_end_sync+0x36/0x70
Jul 12 17:57:22 mindpipe kernel:  [__commit_trans_jl+223/240] __commit_trans_jl+0xdf/0xf0
Jul 12 17:57:22 mindpipe kernel:  [reiserfs_commit_for_inode+32/64] reiserfs_commit_for_inode+0x20/0x40
Jul 12 17:57:22 mindpipe kernel:  [reiserfs_sync_file+77/144] reiserfs_sync_file+0x4d/0x90
Jul 12 17:57:22 mindpipe kernel:  [sys_fsync+123/176] sys_fsync+0x7b/0xb0
Jul 12 17:57:22 mindpipe kernel:  [syscall_call+7/11] syscall_call+0x7/0xb
Jul 12 17:57:22 mindpipe kernel:
Jul 12 17:57:22 mindpipe kernel:
Jul 12 17:57:22 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 17:57:22 mindpipe kernel:  [__crc_totalram_pages+1387264/5353478] snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 17:57:22 mindpipe kernel:  [__crc_totalram_pages+1455070/5353478] snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 17:57:22 mindpipe kernel:  [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 17:57:22 mindpipe kernel:  [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 17:57:22 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 17:57:22 mindpipe kernel:  [remove_journal_hash+99/208] remove_journal_hash+0x63/0xd0
Jul 12 17:57:22 mindpipe kernel:  [remove_all_from_journal_list+78/160] remove_all_from_journal_list+0x4e/0xa0
Jul 12 17:57:22 mindpipe kernel:  [flush_journal_list+496/1344] flush_journal_list+0x1f0/0x540
Jul 12 17:57:22 mindpipe kernel:  [flush_older_journal_lists+48/64] flush_older_journal_lists+0x30/0x40
Jul 12 17:57:22 mindpipe kernel:  [flush_journal_list+795/1344] flush_journal_list+0x31b/0x540
Jul 12 17:57:22 mindpipe kernel:  [flush_used_journal_lists+147/160] flush_used_journal_lists+0x93/0xa0
Jul 12 17:57:22 mindpipe kernel:  [flush_old_journal_lists+69/112] flush_old_journal_lists+0x45/0x70
Jul 12 17:57:22 mindpipe kernel:  [do_journal_end+1853/2848] do_journal_end+0x73d/0xb20
Jul 12 17:57:22 mindpipe kernel:  [journal_end_sync+54/112] journal_end_sync+0x36/0x70
Jul 12 17:57:22 mindpipe kernel:  [__commit_trans_jl+223/240] __commit_trans_jl+0xdf/0xf0
Jul 12 17:57:22 mindpipe kernel:  [reiserfs_commit_for_inode+32/64] reiserfs_commit_for_inode+0x20/0x40
Jul 12 17:57:22 mindpipe kernel:  [reiserfs_sync_file+77/144] reiserfs_sync_file+0x4d/0x90
Jul 12 17:57:22 mindpipe kernel:  [sys_fsync+123/176] sys_fsync+0x7b/0xb0
Jul 12 17:57:22 mindpipe kernel:  [syscall_call+7/11] syscall_call+0x7/0xb
Jul 12 17:57:22 mindpipe kernel:
Jul 12 17:57:22 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 17:57:22 mindpipe kernel:  [__crc_totalram_pages+1387264/5353478] snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 17:57:22 mindpipe kernel:  [__crc_totalram_pages+1455070/5353478] snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 17:57:22 mindpipe kernel:  [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 17:57:22 mindpipe kernel:  [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 17:57:22 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 17:57:22 mindpipe kernel:  [remove_journal_hash+72/208] remove_journal_hash+0x48/0xd0
Jul 12 17:57:22 mindpipe kernel:  [remove_all_from_journal_list+78/160] remove_all_from_journal_list+0x4e/0xa0
Jul 12 17:57:22 mindpipe kernel:  [flush_journal_list+496/1344] flush_journal_list+0x1f0/0x540
Jul 12 17:57:22 mindpipe kernel:  [flush_older_journal_lists+48/64] flush_older_journal_lists+0x30/0x40
Jul 12 17:57:22 mindpipe kernel:  [flush_journal_list+795/1344] flush_journal_list+0x31b/0x540
Jul 12 17:57:22 mindpipe kernel:  [flush_used_journal_lists+147/160] flush_used_journal_lists+0x93/0xa0
Jul 12 17:57:22 mindpipe kernel:  [flush_old_journal_lists+69/112] flush_old_journal_lists+0x45/0x70
Jul 12 17:57:22 mindpipe kernel:  [do_journal_end+1853/2848] do_journal_end+0x73d/0xb20
Jul 12 17:57:22 mindpipe kernel:  [journal_end_sync+54/112] journal_end_sync+0x36/0x70
Jul 12 17:57:22 mindpipe kernel:  [__commit_trans_jl+223/240] __commit_trans_jl+0xdf/0xf0
Jul 12 17:57:22 mindpipe kernel:  [reiserfs_commit_for_inode+32/64] reiserfs_commit_for_inode+0x20/0x40
Jul 12 17:57:22 mindpipe kernel:  [reiserfs_sync_file+77/144] reiserfs_sync_file+0x4d/0x90
Jul 12 17:57:22 mindpipe kernel:  [sys_fsync+123/176] sys_fsync+0x7b/0xb0
Jul 12 17:57:22 mindpipe kernel:  [syscall_call+7/11] syscall_call+0x7/0xb
Jul 12 17:57:22 mindpipe kernel:
Jul 12 17:57:22 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 17:57:22 mindpipe kernel:  [__crc_totalram_pages+1387264/5353478] snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 17:57:22 mindpipe kernel:  [__crc_totalram_pages+1455070/5353478] snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 17:57:22 mindpipe kernel:  [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 17:57:22 mindpipe kernel:  [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 17:57:22 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 17:57:22 mindpipe kernel:  [remove_journal_hash+72/208] remove_journal_hash+0x48/0xd0
Jul 12 17:57:22 mindpipe kernel:  [remove_all_from_journal_list+78/160] remove_all_from_journal_list+0x4e/0xa0
Jul 12 17:57:22 mindpipe kernel:  [flush_journal_list+496/1344] flush_journal_list+0x1f0/0x540
Jul 12 17:57:22 mindpipe kernel:  [flush_older_journal_lists+48/64] flush_older_journal_lists+0x30/0x40
Jul 12 17:57:22 mindpipe kernel:  [flush_journal_list+795/1344] flush_journal_list+0x31b/0x540
Jul 12 17:57:22 mindpipe kernel:  [flush_used_journal_lists+147/160] flush_used_journal_lists+0x93/0xa0
Jul 12 17:57:22 mindpipe kernel:  [flush_old_journal_lists+69/112] flush_old_journal_lists+0x45/0x70
Jul 12 17:57:22 mindpipe kernel:  [do_journal_end+1853/2848] do_journal_end+0x73d/0xb20
Jul 12 17:57:22 mindpipe kernel:  [journal_end_sync+54/112] journal_end_sync+0x36/0x70
Jul 12 17:57:22 mindpipe kernel:  [__commit_trans_jl+223/240] __commit_trans_jl+0xdf/0xf0
Jul 12 17:57:22 mindpipe kernel:  [reiserfs_commit_for_inode+32/64] reiserfs_commit_for_inode+0x20/0x40
Jul 12 17:57:22 mindpipe kernel:  [reiserfs_sync_file+77/144] reiserfs_sync_file+0x4d/0x90
Jul 12 17:57:22 mindpipe kernel:  [sys_fsync+123/176] sys_fsync+0x7b/0xb0
Jul 12 17:57:22 mindpipe kernel:  [syscall_call+7/11] syscall_call+0x7/0xb
Jul 12 17:57:22 mindpipe kernel:
Jul 12 17:57:22 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 17:57:22 mindpipe kernel:  [__crc_totalram_pages+1387264/5353478] snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 17:57:22 mindpipe kernel:  [__crc_totalram_pages+1455070/5353478] snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 17:57:22 mindpipe kernel:  [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 17:57:22 mindpipe kernel:  [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 17:57:22 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 17:57:22 mindpipe kernel:  [remove_journal_hash+67/208] remove_journal_hash+0x43/0xd0
Jul 12 17:57:22 mindpipe kernel:  [remove_all_from_journal_list+78/160] remove_all_from_journal_list+0x4e/0xa0
Jul 12 17:57:22 mindpipe kernel:  [flush_journal_list+496/1344] flush_journal_list+0x1f0/0x540
Jul 12 17:57:22 mindpipe kernel:  [flush_older_journal_lists+48/64] flush_older_journal_lists+0x30/0x40
Jul 12 17:57:22 mindpipe kernel:  [flush_journal_list+795/1344] flush_journal_list+0x31b/0x540
Jul 12 17:57:22 mindpipe kernel:  [flush_used_journal_lists+147/160] flush_used_journal_lists+0x93/0xa0
Jul 12 17:57:22 mindpipe kernel:  [flush_old_journal_lists+69/112] flush_old_journal_lists+0x45/0x70
Jul 12 17:57:22 mindpipe kernel:  [do_journal_end+1853/2848] do_journal_end+0x73d/0xb20
Jul 12 17:57:22 mindpipe kernel:  [journal_end_sync+54/112] journal_end_sync+0x36/0x70
Jul 12 17:57:22 mindpipe kernel:  [__commit_trans_jl+223/240] __commit_trans_jl+0xdf/0xf0
Jul 12 17:57:22 mindpipe kernel:  [reiserfs_commit_for_inode+32/64] reiserfs_commit_for_inode+0x20/0x40
Jul 12 17:57:22 mindpipe kernel:  [reiserfs_sync_file+77/144] reiserfs_sync_file+0x4d/0x90
Jul 12 17:57:22 mindpipe kernel:  [sys_fsync+123/176] sys_fsync+0x7b/0xb0
Jul 12 17:57:22 mindpipe kernel:  [syscall_call+7/11] syscall_call+0x7/0xb
Jul 12 17:57:22 mindpipe kernel:
Jul 12 17:57:22 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 17:57:22 mindpipe kernel:  [__crc_totalram_pages+1387264/5353478] snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 17:57:22 mindpipe kernel:  [__crc_totalram_pages+1455070/5353478] snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 17:57:22 mindpipe kernel:  [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 17:57:22 mindpipe kernel:  [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 17:57:22 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 17:57:22 mindpipe kernel:  [remove_journal_hash+108/208] remove_journal_hash+0x6c/0xd0
Jul 12 17:57:22 mindpipe kernel:  [remove_all_from_journal_list+78/160] remove_all_from_journal_list+0x4e/0xa0
Jul 12 17:57:22 mindpipe kernel:  [flush_journal_list+496/1344] flush_journal_list+0x1f0/0x540
Jul 12 17:57:22 mindpipe kernel:  [flush_older_journal_lists+48/64] flush_older_journal_lists+0x30/0x40
Jul 12 17:57:22 mindpipe kernel:  [flush_journal_list+795/1344] flush_journal_list+0x31b/0x540
Jul 12 17:57:22 mindpipe kernel:  [flush_used_journal_lists+147/160] flush_used_journal_lists+0x93/0xa0
Jul 12 17:57:22 mindpipe kernel:  [flush_old_journal_lists+69/112] flush_old_journal_lists+0x45/0x70
Jul 12 17:57:22 mindpipe kernel:  [do_journal_end+1853/2848] do_journal_end+0x73d/0xb20
Jul 12 17:57:22 mindpipe kernel:  [journal_end_sync+54/112] journal_end_sync+0x36/0x70
Jul 12 17:57:22 mindpipe kernel:  [__commit_trans_jl+223/240] __commit_trans_jl+0xdf/0xf0
Jul 12 17:57:22 mindpipe kernel:  [reiserfs_commit_for_inode+32/64] reiserfs_commit_for_inode+0x20/0x40
Jul 12 17:57:22 mindpipe kernel:  [reiserfs_sync_file+77/144] reiserfs_sync_file+0x4d/0x90
Jul 12 17:57:22 mindpipe kernel:  [sys_fsync+123/176] sys_fsync+0x7b/0xb0
Jul 12 17:57:22 mindpipe kernel:  [syscall_call+7/11] syscall_call+0x7/0xb Jul 12 17:57:22 mindpipe kernel:
Jul 12 17:57:22 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 17:57:22 mindpipe kernel:  [__crc_totalram_pages+1387264/5353478] snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 17:57:22 mindpipe kernel:  [__crc_totalram_pages+1455070/5353478] snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 17:57:22 mindpipe kernel:  [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 17:57:22 mindpipe kernel:  [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 17:57:22 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 17:57:22 mindpipe kernel:  [remove_journal_hash+72/208] remove_journal_hash+0x48/0xd0
Jul 12 17:57:22 mindpipe kernel:  [remove_all_from_journal_list+78/160] remove_all_from_journal_list+0x4e/0xa0
Jul 12 17:57:22 mindpipe kernel:  [flush_journal_list+496/1344] flush_journal_list+0x1f0/0x540
Jul 12 17:57:22 mindpipe kernel:  [flush_older_journal_lists+48/64] flush_older_journal_lists+0x30/0x40
Jul 12 17:57:22 mindpipe kernel:  [flush_journal_list+795/1344] flush_journal_list+0x31b/0x540 Jul 12 17:57:22 mindpipe kernel:  [flush_used_journal_lists+147/160] flush_used_journal_lists+0x93/0xa0
Jul 12 17:57:22 mindpipe kernel:  [flush_old_journal_lists+69/112] flush_old_journal_lists+0x45/0x70
Jul 12 17:57:22 mindpipe kernel:  [do_journal_end+1853/2848] do_journal_end+0x73d/0xb20
Jul 12 17:57:22 mindpipe kernel:  [journal_end_sync+54/112] journal_end_sync+0x36/0x70
Jul 12 17:57:22 mindpipe kernel:  [__commit_trans_jl+223/240] __commit_trans_jl+0xdf/0xf0
Jul 12 17:57:22 mindpipe kernel:  [reiserfs_commit_for_inode+32/64] reiserfs_commit_for_inode+0x20/0x40
Jul 12 17:57:22 mindpipe kernel:  [reiserfs_sync_file+77/144] reiserfs_sync_file+0x4d/0x90
Jul 12 17:57:22 mindpipe kernel:  [sys_fsync+123/176] sys_fsync+0x7b/0xb0
Jul 12 17:57:22 mindpipe kernel:  [syscall_call+7/11] syscall_call+0x7/0xb
Jul 12 17:57:22 mindpipe kernel:
Jul 12 17:57:22 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 17:57:22 mindpipe kernel:  [__crc_totalram_pages+1387264/5353478] snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 17:57:22 mindpipe kernel:  [__crc_totalram_pages+1455070/5353478] snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1] Jul 12 17:57:22 mindpipe kernel:  [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 17:57:22 mindpipe kernel:  [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 17:57:22 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 17:57:22 mindpipe kernel:  [generic_unplug_device+32/96] generic_unplug_device+0x20/0x60
Jul 12 17:57:22 mindpipe kernel:  [blk_backing_dev_unplug+35/48] blk_backing_dev_unplug+0x23/0x30
Jul 12 17:57:22 mindpipe kernel:  [sync_buffer+53/64] sync_buffer+0x35/0x40
Jul 12 17:57:22 mindpipe kernel:  [__wait_on_buffer+150/160] __wait_on_buffer+0x96/0xa0
Jul 12 17:57:22 mindpipe kernel:  [bh_wake_function+0/64] bh_wake_function+0x0/0x40
Jul 12 17:57:22 mindpipe kernel:  [bh_wake_function+0/64] bh_wake_function+0x0/0x40
Jul 12 17:57:22 mindpipe kernel:  [_update_journal_header_block+177/240] _update_journal_header_block+0xb1/0xf0
Jul 12 17:57:22 mindpipe kernel:  [update_journal_header_block+21/48] update_journal_header_block+0x15/0x30
Jul 12 17:57:22 mindpipe kernel:  [flush_journal_list+835/1344] flush_journal_list+0x343/0x540 Jul 12 17:57:22 mindpipe kernel:  [flush_used_journal_lists+147/160] flush_used_journal_lists+0x93/0xa0
Jul 12 17:57:22 mindpipe kernel:  [flush_old_journal_lists+69/112] flush_old_journal_lists+0x45/0x70
Jul 12 17:57:22 mindpipe kernel:  [do_journal_end+1853/2848] do_journal_end+0x73d/0xb20
Jul 12 17:57:22 mindpipe kernel:  [journal_end_sync+54/112] journal_end_sync+0x36/0x70
Jul 12 17:57:22 mindpipe kernel:  [__commit_trans_jl+223/240] __commit_trans_jl+0xdf/0xf0
Jul 12 17:57:22 mindpipe kernel:  [reiserfs_commit_for_inode+32/64] reiserfs_commit_for_inode+0x20/0x40
Jul 12 17:57:22 mindpipe kernel:  [reiserfs_sync_file+77/144] reiserfs_sync_file+0x4d/0x90
Jul 12 17:57:22 mindpipe kernel:  [sys_fsync+123/176] sys_fsync+0x7b/0xb0
Jul 12 17:57:22 mindpipe kernel:  [syscall_call+7/11] syscall_call+0x7/0xb

Hopefully this is not too much information.  You did ask for details...

Lee Revell
Mindpipe Audio

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Andrew Morton]

Lee Revell <rlrevell@joe-job.com> wrote:
>
> On Sun, 2004-07-11 at 01:25, Andrew Morton wrote:
> > What we need to do is to encourage audio testers to use ALSA drivers, to
> > enable CONFIG_SND_DEBUG in the kernel build and to set
> > /proc/asound/*/*/xrun_debug and to send us the traces which result from
> > underruns.
> > 
> 
> OK, here goes.  The following traces result from running JACK overnight
> like so, on an otherwise idle system.  Hardware is a VIA EPIA 6000, with
> a 600Mhz C3 processor.  Kernel is 2.6.7 + volunatary_preempt patch. 
> voluntary_preempt and kernel_preemption are both on.
> 
> jackd -v --realtime -d alsa --outchannels 2 --rate 48000 --shorts
> --playback --period 32  --nperiods 2
> 
> These settings require less than 666 microseconds scheduler latency. 
> The average performance is quite good - 5-20 *microseconds*!

OK, thanks.  The problem areas there are the timer-based route cache
flushing and reiserfs.

We can probably fix the route caceh thing by rescheduling the timer after
having handled 1000 routes or whatever, although I do wonder if this is a
thing we really need to bother about - what else was that machine up to?

resierfs: yes, it's a problem.  I "fixed" it multiple times in 2.4, but the
fixes ended up breaking the fs in subtle ways and I eventually gave up.

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Paul Davis]

>resierfs: yes, it's a problem.  I "fixed" it multiple times in 2.4, but the
>fixes ended up breaking the fs in subtle ways and I eventually gave up.

andrew, this is really helpful. should we conclude that until some
announcement from reiser that they have addressed this, the reiserfs
should be avoided on low latency systems?

--p

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Andrew Morton]

Paul Davis <paul@linuxaudiosystems.com> wrote:
>
> >resierfs: yes, it's a problem.  I "fixed" it multiple times in 2.4, but the
> >fixes ended up breaking the fs in subtle ways and I eventually gave up.
> 
> andrew, this is really helpful. should we conclude that until some
> announcement from reiser that they have addressed this, the reiserfs
> should be avoided on low latency systems?
> 

It seems that way, yes.  I do not know how common the holdoffs are in real
life.  It would be interesting if there was a user report that switching
from reiserfs to ext2/ext3 actually made a difference - this would tell us
that it is indeed a real-world problem.

Note that this info because available because someone set
/proc/asound/*/*/xrun_debug.  We need more people doing that.

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Lee Revell]

On Mon, 2004-07-12 at 20:06, Andrew Morton wrote:
> Paul Davis <paul@linuxaudiosystems.com> wrote:
> >
> > >resierfs: yes, it's a problem.  I "fixed" it multiple times in 2.4, but the
> > >fixes ended up breaking the fs in subtle ways and I eventually gave up.
> > 
> > andrew, this is really helpful. should we conclude that until some
> > announcement from reiser that they have addressed this, the reiserfs
> > should be avoided on low latency systems?
> > 
> 
> It seems that way, yes.  I do not know how common the holdoffs are in real
> life.  It would be interesting if there was a user report that switching
> from reiserfs to ext2/ext3 actually made a difference - this would tell us
> that it is indeed a real-world problem.
> 

This was not a synthetic benchmark, I would consider this a 'real-world'
problem now.  Repeating the test with ext3 would just tell you whether
it has the same problem.

If it is neccesary to get the reiserfs issue addressed, I will repeat
the test with an ext3 system in the next few days,  I would like to hear
from reiser on this before doing much more.

> Note that this info because available because someone set
> /proc/asound/*/*/xrun_debug.  We need more people doing that.
> -

This goes back to the need for ALSA documentation.  Someone needs to
write some.  This will probably require paying that person.  Hopefully
SuSe is working on this, though I suspect I would have heard something.

> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Takashi Iwai]

At Mon, 12 Jul 2004 21:01:16 -0400,
Lee Revell wrote:
> 
> > Note that this info because available because someone set
> > /proc/asound/*/*/xrun_debug.  We need more people doing that.
> > -
> 
> This goes back to the need for ALSA documentation.  Someone needs to
> write some.  This will probably require paying that person.  Hopefully
> SuSe is working on this, though I suspect I would have heard something.

Documentation/sound/alsa/Procfile.txt


Takashi

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Lee Revell]

On Tue, 2004-07-13 at 03:49, Takashi Iwai wrote:
> At Mon, 12 Jul 2004 21:01:16 -0400,
> Lee Revell wrote:
> > 
> > > Note that this info because available because someone set
> > > /proc/asound/*/*/xrun_debug.  We need more people doing that.
> > > -
> > 
> > This goes back to the need for ALSA documentation.  Someone needs to
> > write some.  This will probably require paying that person.  Hopefully
> > SuSe is working on this, though I suspect I would have heard something.
> 
> Documentation/sound/alsa/Procfile.txt
> 

I should have been more specific, I mean user-level documentation.  All
the necessary information for developers is certainly out there, your
driver guide for example is excellent.  I also didn't mean to imply that
the ALSA developers should do it, they are busy enough already...

Lee

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Paul Davis]

>OK, thanks.  The problem areas there are the timer-based route cache
>flushing and reiserfs.
>
>We can probably fix the route caceh thing by rescheduling the timer after
>having handled 1000 routes or whatever, although I do wonder if this is a
>thing we really need to bother about - what else was that machine up to?

i have one concern about this that i talked to takashi about when we
were in bordeaux. it seems to me that the ALSA xrun debug stuff is
measuring things when the interrupt handler for the ALSA device
executes and detects an xrun. if the handler itself was delayed, then
the stack trace for its execution doesn't or might not show what
caused the delay. this means, perhaps, that we need to be rather
careful interpreting these traces.

--p

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Andrew Morton]

Paul Davis <paul@linuxaudiosystems.com> wrote:
>
> >OK, thanks.  The problem areas there are the timer-based route cache
> >flushing and reiserfs.
> >
> >We can probably fix the route caceh thing by rescheduling the timer after
> >having handled 1000 routes or whatever, although I do wonder if this is a
> >thing we really need to bother about - what else was that machine up to?
> 
> i have one concern about this that i talked to takashi about when we
> were in bordeaux. it seems to me that the ALSA xrun debug stuff is
> measuring things when the interrupt handler for the ALSA device
> executes and detects an xrun. if the handler itself was delayed, then
> the stack trace for its execution doesn't or might not show what
> caused the delay. this means, perhaps, that we need to be rather
> careful interpreting these traces.
> 

We can usually tell that from the trace.  If it points up into a busy piece
of code then it's pretty obvious what's happening.  If the trace is due to
a long irq-off time then it will point up into the offending
local_irq_enable().

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Andrea Arcangeli]

On Mon, Jul 12, 2004 at 05:08:44PM -0700, Andrew Morton wrote:
> of code then it's pretty obvious what's happening.  If the trace is due to
> a long irq-off time then it will point up into the offending
> local_irq_enable().

schedule should be called with irq enabled, and I noticed here it's not
(jnz work_resched is executed with irq off so there is a window for
schedule to be called with irq off):

Index: linux-2.5/arch/i386/kernel/entry.S
===================================================================
RCS file: /home/andrea/crypto/cvs/linux-2.5/arch/i386/kernel/entry.S,v
retrieving revision 1.86
diff -u -p -r1.86 entry.S
--- linux-2.5/arch/i386/kernel/entry.S	23 May 2004 05:03:15 -0000	1.86
+++ linux-2.5/arch/i386/kernel/entry.S	13 Jul 2004 04:21:55 -0000
@@ -302,6 +310,7 @@ work_pending:
 	testb $_TIF_NEED_RESCHED, %cl
 	jz work_notifysig
 work_resched:
+	sti
 	call schedule
 	cli				# make sure we don't miss an interrupt
 					# setting need_resched or sigpending

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Lee Revell]

On Tue, 2004-07-13 at 12:25, Andrea Arcangeli wrote:
> On Mon, Jul 12, 2004 at 05:08:44PM -0700, Andrew Morton wrote:
> > of code then it's pretty obvious what's happening.  If the trace is due to
> > a long irq-off time then it will point up into the offending
> > local_irq_enable().
> 
> schedule should be called with irq enabled, and I noticed here it's not
> (jnz work_resched is executed with irq off so there is a window for
> schedule to be called with irq off):
> 
> Index: linux-2.5/arch/i386/kernel/entry.S
> ===================================================================
> RCS file: /home/andrea/crypto/cvs/linux-2.5/arch/i386/kernel/entry.S,v
> retrieving revision 1.86
> diff -u -p -r1.86 entry.S
> --- linux-2.5/arch/i386/kernel/entry.S	23 May 2004 05:03:15 -0000	1.86
> +++ linux-2.5/arch/i386/kernel/entry.S	13 Jul 2004 04:21:55 -0000
> @@ -302,6 +310,7 @@ work_pending:
>  	testb $_TIF_NEED_RESCHED, %cl
>  	jz work_notifysig
>  work_resched:
> +	sti
>  	call schedule
>  	cli				# make sure we don't miss an interrupt
>  					# setting need_resched or sigpending

Would this explain these?  When running JACK with settings that need
sub-millisecond latencies, I get them when I generate any load at all on
the system (typing, switching windows, etc).  I also get lots of these
if I run JACK from an X terminal, but very few if I run it from a text
console, even if X is running in the background.

Jul 13 14:36:16 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:199: Unexpected hw_pointer value [1] (stream = 0, delta: -25, max jitter = 32): wrong interrupt acknowledge?
Jul 13 14:36:16 mindpipe kernel:  [__crc_totalram_pages+1386981/5353478] snd_pcm_period_elapsed+0x1af/0x410 [snd_pcm]
Jul 13 14:36:16 mindpipe kernel:  [__crc_totalram_pages+1455070/5353478] snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 13 14:36:16 mindpipe kernel:  [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 13 14:36:16 mindpipe kernel:  [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 13 14:36:16 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 13 14:36:16 mindpipe kernel:  [__do_softirq+48/144] __do_softirq+0x30/0x90
Jul 13 14:36:16 mindpipe kernel:  [do_softirq+53/64] do_softirq+0x35/0x40
Jul 13 14:36:16 mindpipe kernel:  [do_IRQ+261/352] do_IRQ+0x105/0x160
Jul 13 14:36:16 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 13 14:36:16 mindpipe kernel:
Jul 13 14:36:16 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:199: Unexpected hw_pointer value [1] (stream = 0, delta: -31, max jitter = 32): wrong interrupt acknowledge?
Jul 13 14:36:16 mindpipe kernel:  [__crc_totalram_pages+1386981/5353478] snd_pcm_period_elapsed+0x1af/0x410 [snd_pcm]
Jul 13 14:36:16 mindpipe kernel:  [__crc_totalram_pages+1455070/5353478] snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 13 14:36:16 mindpipe kernel:  [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 13 14:36:16 mindpipe kernel:  [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 13 14:36:16 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 13 14:36:16 mindpipe kernel:  [dnotify_parent+42/160] dnotify_parent+0x2a/0xa0
Jul 13 14:36:16 mindpipe kernel:  [vfs_read+206/256] vfs_read+0xce/0x100
Jul 13 14:36:16 mindpipe kernel:  [sys_read+45/80] sys_read+0x2d/0x50
Jul 13 14:36:16 mindpipe kernel:  [syscall_call+7/11] syscall_call+0x7/0xb
Jul 13 14:36:16 mindpipe kernel:

Lee

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Andrew Morton]

Lee Revell <rlrevell@joe-job.com> wrote:
>
> Would this explain these?  When running JACK with settings that need
> sub-millisecond latencies, I get them when I generate any load at all on
> the system (typing, switching windows, etc).  I also get lots of these
> if I run JACK from an X terminal, but very few if I run it from a text
> console, even if X is running in the background.
> 
> Jul 13 14:36:16 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:199: Unexpected hw_pointer value [1] (stream = 0, delta: -25, max jitter = 32): wrong interrupt acknowledge?

I'm wondering what this message actually means.  "Unexpected hw_pointer
value"?

Does this actually indicate an underrun, or is the debug code screwy?

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Lee Revell]

On Tue, 2004-07-13 at 17:29, Andrew Morton wrote:
> Lee Revell <rlrevell@joe-job.com> wrote:
> >
> > Would this explain these?  When running JACK with settings that need
> > sub-millisecond latencies, I get them when I generate any load at all on
> > the system (typing, switching windows, etc).  I also get lots of these
> > if I run JACK from an X terminal, but very few if I run it from a text
> > console, even if X is running in the background.
> > 
> > Jul 13 14:36:16 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:199: Unexpected hw_pointer value [1] (stream = 0, delta: -25, max jitter = 32): wrong interrupt acknowledge?
> 
> I'm wondering what this message actually means.  "Unexpected hw_pointer
> value"?
> 
> Does this actually indicate an underrun, or is the debug code screwy?

Not sure.  Here is what Takashi had to say about it:

"The message appears when an unexpected DMA pointer is read in the
interrupt handler.  Either the handling of irq was delayed more than
the buffer size, an irq is issued at the wrong timing, or the DMA
pointer reigster is somehow screwed up.

Since you're using quite small buffer, I guess the former case."

My response:

"I thought this was what an XRUN was, when the handling of the irq is
delayed more than the buffer size.  Sometimes these messages are
associated with XRUNs, sometimes not."

Haven't heard back yet. 

Is it possible that I am simply pushing my hardware past its limits? 
Keep in mind this is a 600Mhz C3 processor.

Lee

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Takashi Iwai]

At Tue, 13 Jul 2004 17:45:30 -0400,
Lee Revell wrote:
> 
> On Tue, 2004-07-13 at 17:29, Andrew Morton wrote:
> > Lee Revell <rlrevell@joe-job.com> wrote:
> > >
> > > Would this explain these?  When running JACK with settings that need
> > > sub-millisecond latencies, I get them when I generate any load at all on
> > > the system (typing, switching windows, etc).  I also get lots of these
> > > if I run JACK from an X terminal, but very few if I run it from a text
> > > console, even if X is running in the background.
> > > 
> > > Jul 13 14:36:16 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:199: Unexpected hw_pointer value [1] (stream = 0, delta: -25, max jitter = 32): wrong interrupt acknowledge?
> > 
> > I'm wondering what this message actually means.  "Unexpected hw_pointer
> > value"?
> > 
> > Does this actually indicate an underrun, or is the debug code screwy?
> 
> Not sure.  Here is what Takashi had to say about it:
> 
> "The message appears when an unexpected DMA pointer is read in the
> interrupt handler.  Either the handling of irq was delayed more than
> the buffer size, an irq is issued at the wrong timing, or the DMA
> pointer reigster is somehow screwed up.
> 
> Since you're using quite small buffer, I guess the former case."
> 
> My response:
> 
> "I thought this was what an XRUN was, when the handling of the irq is
> delayed more than the buffer size.  Sometimes these messages are
> associated with XRUNs, sometimes not."
> 
> Haven't heard back yet. 
> 
> Is it possible that I am simply pushing my hardware past its limits? 
> Keep in mind this is a 600Mhz C3 processor.

I think yes.  32 frames / 44.1kHz = 0.725 ms.


Takashi

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Lee Revell]

On Wed, 2004-07-14 at 04:51, Takashi Iwai wrote:
> At Tue, 13 Jul 2004 17:45:30 -0400,
> Lee Revell wrote:
> > 
> > On Tue, 2004-07-13 at 17:29, Andrew Morton wrote:
> > > Lee Revell <rlrevell@joe-job.com> wrote:
> > > >
> > > > Would this explain these?  When running JACK with settings that need
> > > > sub-millisecond latencies, I get them when I generate any load at all on
> > > > the system (typing, switching windows, etc).  I also get lots of these
> > > > if I run JACK from an X terminal, but very few if I run it from a text
> > > > console, even if X is running in the background.
> > > > 
> > > > Jul 13 14:36:16 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:199: Unexpected hw_pointer value [1] (stream = 0, delta: -25, max jitter = 32): wrong interrupt acknowledge?
> > > 
> > > I'm wondering what this message actually means.  "Unexpected hw_pointer
> > > value"?
> > > 
> > > Does this actually indicate an underrun, or is the debug code screwy?
> > 
> > Not sure.  Here is what Takashi had to say about it:
> > 
> > "The message appears when an unexpected DMA pointer is read in the
> > interrupt handler.  Either the handling of irq was delayed more than
> > the buffer size, an irq is issued at the wrong timing, or the DMA
> > pointer reigster is somehow screwed up.
> > 
> > Since you're using quite small buffer, I guess the former case."
> > 
> > My response:
> > 
> > "I thought this was what an XRUN was, when the handling of the irq is
> > delayed more than the buffer size.  Sometimes these messages are
> > associated with XRUNs, sometimes not."
> > 
> > Haven't heard back yet. 
> > 
> > Is it possible that I am simply pushing my hardware past its limits? 
> > Keep in mind this is a 600Mhz C3 processor.
> 
> I think yes.  32 frames / 44.1kHz = 0.725 ms.
> 

I am runnign at 48kHz so it's actually 0.666 ms.  But, the average
response is quite good, 20-30 microseconds.  The spikes are infrequent
enought that I think this is achievable.  If not then 64 frames
definitely is.

So what is the difference between the above message and an XRUN?  I
thought an XRUN occurred when the handling of the IRQ is delayed more
than the buffer size.

Lee

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Takashi Iwai]

At Wed, 14 Jul 2004 05:36:31 -0400,
Lee Revell wrote:
> 
> On Wed, 2004-07-14 at 04:51, Takashi Iwai wrote:
> > At Tue, 13 Jul 2004 17:45:30 -0400,
> > Lee Revell wrote:
> > > 
> > > On Tue, 2004-07-13 at 17:29, Andrew Morton wrote:
> > > > Lee Revell <rlrevell@joe-job.com> wrote:
> > > > >
> > > > > Would this explain these?  When running JACK with settings that need
> > > > > sub-millisecond latencies, I get them when I generate any load at all on
> > > > > the system (typing, switching windows, etc).  I also get lots of these
> > > > > if I run JACK from an X terminal, but very few if I run it from a text
> > > > > console, even if X is running in the background.
> > > > > 
> > > > > Jul 13 14:36:16 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:199: Unexpected hw_pointer value [1] (stream = 0, delta: -25, max jitter = 32): wrong interrupt acknowledge?
> > > > 
> > > > I'm wondering what this message actually means.  "Unexpected hw_pointer
> > > > value"?
> > > > 
> > > > Does this actually indicate an underrun, or is the debug code screwy?
> > > 
> > > Not sure.  Here is what Takashi had to say about it:
> > > 
> > > "The message appears when an unexpected DMA pointer is read in the
> > > interrupt handler.  Either the handling of irq was delayed more than
> > > the buffer size, an irq is issued at the wrong timing, or the DMA
> > > pointer reigster is somehow screwed up.
> > > 
> > > Since you're using quite small buffer, I guess the former case."
> > > 
> > > My response:
> > > 
> > > "I thought this was what an XRUN was, when the handling of the irq is
> > > delayed more than the buffer size.  Sometimes these messages are
> > > associated with XRUNs, sometimes not."
> > > 
> > > Haven't heard back yet. 
> > > 
> > > Is it possible that I am simply pushing my hardware past its limits? 
> > > Keep in mind this is a 600Mhz C3 processor.
> > 
> > I think yes.  32 frames / 44.1kHz = 0.725 ms.
> > 
> 
> I am runnign at 48kHz so it's actually 0.666 ms.  But, the average
> response is quite good, 20-30 microseconds.  The spikes are infrequent
> enought that I think this is achievable.  If not then 64 frames
> definitely is.
> 
> So what is the difference between the above message and an XRUN?  I
> thought an XRUN occurred when the handling of the IRQ is delayed more
> than the buffer size.

Yes, XRUN is so.

The message above is not directly related with XRUN.
It appears when an unexpected DMA pointer is returned.  This can
happen e.g. when the irq handler is called at the wrong timing,
or the register returns a bogus value.


Takashi

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Lee Revell]

On Wed, 2004-07-14 at 06:48, Takashi Iwai wrote:
> At Wed, 14 Jul 2004 05:36:31 -0400,
> Lee Revell wrote:
> > 
> > On Wed, 2004-07-14 at 04:51, Takashi Iwai wrote:
> > > At Tue, 13 Jul 2004 17:45:30 -0400,
> > > Lee Revell wrote:
> > > > 
> > > > On Tue, 2004-07-13 at 17:29, Andrew Morton wrote:
> > > > > Lee Revell <rlrevell@joe-job.com> wrote:
> > > > > >
> > > > > > Would this explain these?  When running JACK with settings that need
> > > > > > sub-millisecond latencies, I get them when I generate any load at all on
> > > > > > the system (typing, switching windows, etc).  I also get lots of these
> > > > > > if I run JACK from an X terminal, but very few if I run it from a text
> > > > > > console, even if X is running in the background.
> > > > > > 
> > > > > > Jul 13 14:36:16 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:199: Unexpected hw_pointer value [1] (stream = 0, delta: -25, max jitter = 32): wrong interrupt acknowledge?
> > > > > 
> > > > > I'm wondering what this message actually means.  "Unexpected hw_pointer
> > > > > value"?
> > > > > 
> > > > > Does this actually indicate an underrun, or is the debug code screwy?
> > > > 
> > > > Not sure.  Here is what Takashi had to say about it:
> > > > 
> > > > "The message appears when an unexpected DMA pointer is read in the
> > > > interrupt handler.  Either the handling of irq was delayed more than
> > > > the buffer size, an irq is issued at the wrong timing, or the DMA
> > > > pointer reigster is somehow screwed up.
> > > > 
> > > > Since you're using quite small buffer, I guess the former case."
> > > > 
> > > > My response:
> > > > 
> > > > "I thought this was what an XRUN was, when the handling of the irq is
> > > > delayed more than the buffer size.  Sometimes these messages are
> > > > associated with XRUNs, sometimes not."
> > > > 
> > > > Haven't heard back yet. 
> > > > 
> > > > Is it possible that I am simply pushing my hardware past its limits? 
> > > > Keep in mind this is a 600Mhz C3 processor.
> > > 
> > > I think yes.  32 frames / 44.1kHz = 0.725 ms.
> > > 
> > 
> > I am runnign at 48kHz so it's actually 0.666 ms.  But, the average
> > response is quite good, 20-30 microseconds.  The spikes are infrequent
> > enought that I think this is achievable.  If not then 64 frames
> > definitely is.
> > 
> > So what is the difference between the above essage and an XRUN?  I
> > thought an XRUN occurred when the handling of the IRQ is delayed more
> > than the buffer size.
> 
> Yes, XRUN is so.
> 
> The message above is not directly related with XRUN.
> It appears when an unexpected DMA pointer is returned.  This can
> happen e.g. when the irq handler is called at the wrong timing,
> or the register returns a bogus value.
> 

I am still kind of confused.  It sounds like this is more serious than
an XRUN.  What would cause this to happen?

Lee

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Lee Revell]

On Wed, 2004-07-14 at 10:52, Lee Revell wrote:
> On Wed, 2004-07-14 at 06:48, Takashi Iwai wrote:
> > At Wed, 14 Jul 2004 05:36:31 -0400,
> > Lee Revell wrote:
> > > 
> > > On Wed, 2004-07-14 at 04:51, Takashi Iwai wrote:
> > > > At Tue, 13 Jul 2004 17:45:30 -0400,
> > > > Lee Revell wrote:
> > > > > 
> > > > > On Tue, 2004-07-13 at 17:29, Andrew Morton wrote:
> > > > > > Lee Revell <rlrevell@joe-job.com> wrote:
> > > > > > >
> > > > > > > Would this explain these?  When running JACK with settings that need
> > > > > > > sub-millisecond latencies, I get them when I generate any load at all on
> > > > > > > the system (typing, switching windows, etc).  I also get lots of these
> > > > > > > if I run JACK from an X terminal, but very few if I run it from a text
> > > > > > > console, even if X is running in the background.
> > > > > > > 
> > > > > > > Jul 13 14:36:16 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:199: Unexpected hw_pointer value [1] (stream = 0, delta: -25, max jitter = 32): wrong interrupt acknowledge?
> > > > > > 
> > > > > > I'm wondering what this message actually means.  "Unexpected hw_pointer
> > > > > > value"?
> > > > > > 
> > > > > > Does this actually indicate an underrun, or is the debug code screwy?
> > > > > 
> > > > > Not sure.  Here is what Takashi had to say about it:
> > > > > 
> > > > > "The message appears when an unexpected DMA pointer is read in the
> > > > > interrupt handler.  Either the handling of irq was delayed more than
> > > > > the buffer size, an irq is issued at the wrong timing, or the DMA
> > > > > pointer reigster is somehow screwed up.
> > > > > 
> > > > > Since you're using quite small buffer, I guess the former case."
> > > > > 
> > > > > My response:
> > > > > 
> > > > > "I thought this was what an XRUN was, when the handling of the irq is
> > > > > delayed more than the buffer size.  Sometimes these messages are
> > > > > associated with XRUNs, sometimes not."
> > > > > 
> > > > > Haven't heard back yet. 
> > > > > 
> > > > > Is it possible that I am simply pushing my hardware past its limits? 
> > > > > Keep in mind this is a 600Mhz C3 processor.
> > > > 
> > > > I think yes.  32 frames / 44.1kHz = 0.725 ms.
> > > > 
> > > 
> > > I am runnign at 48kHz so it's actually 0.666 ms.  But, the average
> > > response is quite good, 20-30 microseconds.  The spikes are infrequent
> > > enought that I think this is achievable.  If not then 64 frames
> > > definitely is.
> > > 
> > > So what is the difference between the above essage and an XRUN?  I
> > > thought an XRUN occurred when the handling of the IRQ is delayed more
> > > than the buffer size.
> > 
> > Yes, XRUN is so.
> > 
> > The message above is not directly related with XRUN.
> > It appears when an unexpected DMA pointer is returned.  This can
> > happen e.g. when the irq handler is called at the wrong timing,
> > or the register returns a bogus value.
> > 
> 
> I am still kind of confused.  It sounds like this is more serious than
> an XRUN.  What would cause this to happen?
> 

OK, I think I understand the distinction.  An XRUN is when a process is
writing (or reading) to a PCM device, the buffer half empty interrupt
occurs, and the interrupt handler runs, which makes the process ready to
run, but the scheduler does not run it in time to write the next block
of data to the buffer.

The "Unexpected hw_pointer" error would occur when the device generates
an interrupt, but there is a delay in running the interrupt handler, and
by the time the interrupt handler runs, the hardware pointer has moved
somewhere we don't expect it to be.  This would occur if we missed an
interrupt, or several, due to interrupts being disabled for longer than
the period size.  So the distinction is scheduler latency vs. interrupt
response latency.

What is the longest that the kernel disables interrupts?  I remember
hearing 500 microseconds, but this was a while back.

Lee

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Benno Senoner]

Takashi Iwai wrote:

>>Is it possible that I am simply pushing my hardware past its limits? 
>>Keep in mind this is a 600Mhz C3 processor.
>>    
>>
>
>I think yes.  32 frames / 44.1kHz = 0.725 ms.
>  
>
I don't think so, I think it's because the Linux scheduler (and kernel 
in general) since it's not a RTOS
is pushed to the limits. (but as we see it can still be optimized).

For example I used the same VIA box with RTLinux to remote control servo 
motors which need a PWM signal
of the duration of 2msec and based on the location of the negative flank 
(from high to low) the servo
motor goes in a certain position.
For example if the duration of the pulse is 2msec then setting the flank 
at 0msec (at the beginning of the cycle)
the servo goes to -45degrees , 1msec 0degrees , 2msec +45degrees.

Jitter in the pulse can be detected when the servo is vibrating a bit 
around the nominal position.
Of course a very short lived spike cannot be detected by the servo 
because of the motor's inertia
but I tried to put the box under very high load especially video 
playback (the VIA box uses a shared bus architecture
holding the video data in the PC's RAM), HD load etc but the jitter is 
very minimal, probably 30-40usec because the
servos vibrate only about 1degree or so (only when the box is under very 
high load).

This just to say that the VIA box should easily be able to cope with 32 
audio frames (0.6msec buffers) from a hardware
point of view.
Anyway "worst case" latencies (or better latencies under very high load) 
of <0.5-1msec are completely adequate for
real time multimedia because if you shorten your audio processing cycles 
too much (eg 32 frames) then the setup
overhead of DSP routines, and scheduling overhead becomes big and the 
efficiency of
the algorithms decrease quite a bit.
imagine running jack with 10 client applications and 32 frames (0.6msec 
periods) , this means that within 0.6msec you need
to reschedule 10 times = 60 usec per client. I don't know how much the 
actual rescheduling of a process takes (AFAIK around 1usec ?)
but I guess the main problem is the constant invalidation of the cache 
because those audio clients run for a really short time.
Of course if you have only 1-2 clients running then the efficiency at 32 
frames should still be good (but it will certainly provide 10-20% less
perfromance than using 64 or 128 frames).
Our goal should be rock solid operation at 64msec (around 1msec-1.5msec 
processing periods).
If you use 2-3 buffers then the kernel has still another 1.5-3msec of 
headroom before an actual (audible) xrun occurs.

cheers,
Benno
http://www.linuxsampler.org


>
>Takashi
>
>  
>

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Andrew Morton]

Andrea Arcangeli <andrea@suse.de> wrote:
>
> On Mon, Jul 12, 2004 at 05:08:44PM -0700, Andrew Morton wrote:
> > of code then it's pretty obvious what's happening.  If the trace is due to
> > a long irq-off time then it will point up into the offending
> > local_irq_enable().
> 
> schedule should be called with irq enabled, and I noticed here it's not
> (jnz work_resched is executed with irq off so there is a window for
> schedule to be called with irq off):
> 
> Index: linux-2.5/arch/i386/kernel/entry.S
> ===================================================================
> RCS file: /home/andrea/crypto/cvs/linux-2.5/arch/i386/kernel/entry.S,v
> retrieving revision 1.86
> diff -u -p -r1.86 entry.S
> --- linux-2.5/arch/i386/kernel/entry.S	23 May 2004 05:03:15 -0000	1.86
> +++ linux-2.5/arch/i386/kernel/entry.S	13 Jul 2004 04:21:55 -0000
> @@ -302,6 +310,7 @@ work_pending:
>  	testb $_TIF_NEED_RESCHED, %cl
>  	jz work_notifysig
>  work_resched:
> +	sti
>  	call schedule
>  	cli				# make sure we don't miss an interrupt
>  					# setting need_resched or sigpending

sys_sched_yield() also calls schedule() with local interrupts disabled. 
It's a bit grubby, but saves a few cycles.  Nick and Ingo prefer it that way.

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Andrea Arcangeli]

On Tue, Jul 13, 2004 at 11:48:29AM -0700, Andrew Morton wrote:
> sys_sched_yield() also calls schedule() with local interrupts disabled. 
> It's a bit grubby, but saves a few cycles.  Nick and Ingo prefer it that way.

we can remove the irqs_disabled() check in might_sleep then, I'd like to
call might_sleep from cond_resched.

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Andrew Morton]

Andrea Arcangeli <andrea@suse.de> wrote:
>
> On Tue, Jul 13, 2004 at 11:48:29AM -0700, Andrew Morton wrote:
> > sys_sched_yield() also calls schedule() with local interrupts disabled. 
> > It's a bit grubby, but saves a few cycles.  Nick and Ingo prefer it that way.
> 
> we can remove the irqs_disabled() check in might_sleep then, I'd like to
> call might_sleep from cond_resched.

Confused.  Where do we call cond_resched() with local interrupts disabled?

Sleeping with local interrupts disabled is usually a bug, so we should prefer
to keep that check in might_sleep().

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Andrea Arcangeli]

On Tue, Jul 13, 2004 at 02:54:24PM -0700, Andrew Morton wrote:
> Confused.  Where do we call cond_resched() with local interrupts disabled?

there are a lot of cond_resched, we might be calling it with irq
disabled, nobody ever did a might_sleep in the fast path of
cond_resched. And even if nobody does, if entry.S and sched_yield can
optimize, then everybody else can optimize too. It's not like that
entry.S is a piece of scheduler internal that will be obviously modified
if we modify the scheduler. so if you intend to leave those two there's
no point to forbid others to optimize, and there's no might_sleep in
current cond_resched anyways so you're already allowing people to
optimize and I want to allow it still after I add might_sleep there.

> Sleeping with local interrupts disabled is usually a bug, so we should
> prefer to keep that check in might_sleep().

either it's _always_ a bug including for entry.S or sched_yield, or it's
_never_ a bug. I don't understand the "usually".

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Andrew Morton]

Andrea Arcangeli <andrea@suse.de> wrote:
>
> > Sleeping with local interrupts disabled is usually a bug, so we should
> > prefer to keep that check in might_sleep().
> 
> either it's _always_ a bug including for entry.S or sched_yield, or it's
> _never_ a bug. I don't understand the "usually".

If some code does:


	local_irq_disable();
	<fiddle with per-cpu stuff>
	kmalloc(GFP_KERNEL);
	<fiddle with per-cpu stuff>
	local_irq_enable();

or

	local_irq_disable();
	<fiddle with per-cpu stuff>
	function_which_calls_cond_resched();
	<fiddle with per-cpu stuff>
	local_irq_enable();

then we want might_sleep() to warn about the bug.

The fact that a couple of scheduler-internal fastpaths happen to know that
they can call schedule() with interrupts disabled is not relevant to this.

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Andrea Arcangeli]

On Tue, Jul 13, 2004 at 03:25:32PM -0700, Andrew Morton wrote:
> 	local_irq_disable();
> 	<fiddle with per-cpu stuff>
> 	function_which_calls_cond_resched();
> 	<fiddle with per-cpu stuff>
> 	local_irq_enable();
> 
> then we want might_sleep() to warn about the bug.

might_sleep currently _doesn't_ warn about any bug in the above case I
quoted.

the kmalloc example is trapped instead.

>From my part I don't like anybody to call schedule with irq disabled
(and I would definitely put a debug check in schedule() for that, guess
how I found about the missing sti in entry.S btw). But if you are ok
with people calling schedule with irq disabled then I cannot put a check
in there. sti doesn't cost that much, and the work-to-do and sched_yield
paths are _never_ fast paths, so they don't worth an hack like that.

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Andrew Morton]

Andrea Arcangeli <andrea@suse.de> wrote:
>
> On Tue, Jul 13, 2004 at 03:25:32PM -0700, Andrew Morton wrote:
> > 	local_irq_disable();
> > 	<fiddle with per-cpu stuff>
> > 	function_which_calls_cond_resched();
> > 	<fiddle with per-cpu stuff>
> > 	local_irq_enable();
> > 
> > then we want might_sleep() to warn about the bug.
> 
> might_sleep currently _doesn't_ warn about any bug in the above case I
> quoted.
> 
> the kmalloc example is trapped instead.

Yeah, I know.  might_sleep() in cond_resched() makes sense.

> >From my part I don't like anybody to call schedule with irq disabled

I'd agree with that.  But when I "fixed" it, Ingo unfixed it again and I
didn't have sufficiently strong feelings either way to object.

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Andrea Arcangeli]

On Tue, Jul 13, 2004 at 03:44:48PM -0700, Andrew Morton wrote:
> Yeah, I know.  might_sleep() in cond_resched() makes sense.

What I'm doing is basically to replace all might_sleep with cond_resched
and then I add might_sleep in cond_resched. I also merged all
new might_sleep in Ingo's patch (converted to cond_resched). We'll see
what happens then when I try to boot such a thing (the sti and
sched_yield already given me some troubles).

I was considering adding a cond_resched_costly but I didn't see anything
really that costly to need a CONFIG_LOW_RESCHED_OVERHEAD.

btw, cond_resched should only be defined as might_sleep with PREEMPT
enabled, otherwise it's pointless to check need_resched at almost every
spin_unlock and to do it during cond_resched too. if might_sleep doesn't
BUG it means we didn't need to check need_resched in the first place if
preempt is enabled.

cond_resched_lock is another story of course.

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Andrew Morton]

Andrea Arcangeli <andrea@suse.de> wrote:
>
> What I'm doing is basically to replace all might_sleep with cond_resched

I cannot see a lot of point in that.  They are semantically different
things and should look different in the source.

And it's currently OK to add a might_sleep() to (say) an inline path which
is expended a zillion times because we know it'll go away for production
builds.  If those things become cond_resched() calls instead, the code
increase will be permanent.

> cond_resched_lock is another story of course.

cond_resched_lock() doesn't work on SMP.  I'll probably be removing it in
favour of unconditionally dropping the lock every N times around the loop,
to allow the other CPU (the one with need_resched() true) to get in there
and take it.

And please let me repeat: preemption is the way in which we wish to provide
low-latency.  At this time, patches which sprinkle cond_resched() all over
the place are unwelcome.  After 2.7 forks we can look at it again.

I've yet to go through Arjan's patch - I suspect a lot of it is not needed.

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Andrea Arcangeli]

On Tue, Jul 13, 2004 at 04:06:28PM -0700, Andrew Morton wrote:
> Andrea Arcangeli <andrea@suse.de> wrote:
> >
> > What I'm doing is basically to replace all might_sleep with cond_resched
> 
> I cannot see a lot of point in that.  They are semantically different
> things and should look different in the source.
> 
> And it's currently OK to add a might_sleep() to (say) an inline path which
> is expended a zillion times because we know it'll go away for production
> builds.  If those things become cond_resched() calls instead, the code
> increase will be permanent.

this is exactly why I'm making this change: so you can still add
might_sleep in a inline path expected to run a zillion times. With
Ingo's change you would be doing cond_sched internally to might_sleep, I
do the other way around so might_sleep remains a debugging statement.

As Ingo basically showed (and I agree), all current might_sleep seems
suitable to be converted to cond_resched. I checked all them and there's
none that seems to be called in a loop for no good reason. the ones in
the semaphore are quite nice too since it's better to schedule right
before taking the semaphore than after getting it. The one in the
semaphore and in the copy-user are the only place where preempt seems to
be lower overhead but in most other places a spinlock is being taken
very near to the cond_resched.

> > cond_resched_lock is another story of course.
> 
> cond_resched_lock() doesn't work on SMP.  I'll probably be removing it in
> favour of unconditionally dropping the lock every N times around the loop,
> to allow the other CPU (the one with need_resched() true) to get in there
> and take it.
> 
> And please let me repeat: preemption is the way in which we wish to provide
> low-latency.  At this time, patches which sprinkle cond_resched() all over
> the place are unwelcome.  After 2.7 forks we can look at it again.

I think it's a mistake to believe people that leaves preempt off don't
care about lowlatency (this is also why I always keep this change in my
tree)

Index: linux-2.5/mm/memory.c
===================================================================
RCS file: /home/andrea/crypto/cvs/linux-2.5/mm/memory.c,v
retrieving revision 1.178
diff -u -p -r1.178 memory.c
--- linux-2.5/mm/memory.c	30 Jun 2004 02:21:13 -0000	1.178
+++ linux-2.5/mm/memory.c	13 Jul 2004 03:33:45 -0000
@@ -479,20 +479,13 @@ static void unmap_page_range(struct mmu_
 }
 
 /* Dispose of an entire struct mmu_gather per rescheduling point */
-#if defined(CONFIG_SMP) && defined(CONFIG_PREEMPT)
+#if defined(CONFIG_SMP)
 #define ZAP_BLOCK_SIZE	(FREE_PTE_NR * PAGE_SIZE)
-#endif
-
+#else
 /* For UP, 256 pages at a time gives nice low latency */
-#if !defined(CONFIG_SMP) && defined(CONFIG_PREEMPT)
 #define ZAP_BLOCK_SIZE	(256 * PAGE_SIZE)
 #endif
 
-/* No preempt: go for improved straight-line efficiency */
-#if !defined(CONFIG_PREEMPT)
-#define ZAP_BLOCK_SIZE	(1024 * PAGE_SIZE)
-#endif
-
 /**
  * unmap_vmas - unmap a range of memory covered by a list of vma's
  * @tlbp: address of the caller's struct mmu_gather


having a CONFIG_PREEMPT in the above code, means you believe the people
who leaves preempt off don't care about lowlatency and that's really not
true. Ingo's effort as well shows people care about lowlatency even with
preempt off.

The people who leaves preempt off simply thinks they can get the same
lowlatency but with less overhead in the locks (though there will be
cond_resched in the semaphore, but semaphore is less performance
critical than the spinlock normally, and semaphore is usually hold for a
longer period of time, short critical sections would better go with the
spinlock anyways to avoid overscheduling).

> I've yet to go through Arjan's patch - I suspect a lot of it is not needed.

Arjan's or Ingo's? I've seen Ingo's patch but maybe I missed Arjan's one.

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Andrew Morton]

Andrea Arcangeli <andrea@suse.de> wrote:
>
> > And it's currently OK to add a might_sleep() to (say) an inline path which
> > is expended a zillion times because we know it'll go away for production
> > builds.  If those things become cond_resched() calls instead, the code
> > increase will be permanent.
> 
> this is exactly why I'm making this change: so you can still add
> might_sleep in a inline path expected to run a zillion times. With
> Ingo's change you would be doing cond_sched internally to might_sleep, I
> do the other way around so might_sleep remains a debugging statement.

OK.


cond_resched() is usually a waste of space with CONFIG_PREEMPT.  It might
make sense to define a cond_resched_if_not_preempt thingy, which only does
things if !CONFIG_PREEMPT.  We'd still need to use cond_resched() inside
lock_kernel().

> 
> > I've yet to go through Arjan's patch - I suspect a lot of it is not needed.
> 
> Arjan's or Ingo's? I've seen Ingo's patch but maybe I missed Arjan's one.

I think Ingo's patch includes Arjan's one.  Ingo's patch apparently breaks
ext3.  I have a bunch of ext3 and other fixes here, but there's still an occasional
problem on SMP.

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Andrea Arcangeli]

On Tue, Jul 13, 2004 at 04:32:36PM -0700, Andrew Morton wrote:
> OK.
> 
> 
> cond_resched() is usually a waste of space with CONFIG_PREEMPT.  It might
> make sense to define a cond_resched_if_not_preempt thingy, which only does
> things if !CONFIG_PREEMPT.  We'd still need to use cond_resched() inside
> lock_kernel().

Though most of the current cond_resched invocations in mainline (and the
future ones too) should be optimized away with preempt enabled, so maybe
it's simpler to just implement a cond_resched_force() that isn't
optimized away with preempt enabled and use it in lock_kernel? I'm
missing where cond_resched is needed inside lock_kernel though,
preempt_schedule already checks TIF_NEED_RESCHED by hand (on the top of
my mind I don't recall cond_rescheds that shouldn't be optimized away
with preempt enabled).

> I think Ingo's patch includes Arjan's one.  Ingo's patch apparently breaks

Ok.

> ext3.  I have a bunch of ext3 and other fixes here, but there's still an occasional
> problem on SMP.

I also had instability with it and I've seen debug checks in ext3 too
(but it's not finished anyways), and I've still a bit of printk flooding
so I'm uncertain if it's my mistake due the more strict debugging I
added in schedule() or an issue in the newer cond_resched but thanks for
the info (sounds like not all the debug checks triggering are my mistake ;).

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Andrew Morton]

Andrea Arcangeli <andrea@suse.de> wrote:
>
> I'm missing where cond_resched is needed inside lock_kernel though,

Anywhere where we do lots of work inside lock_kernel().  Various ioctls and
ext3 journal recovery are instances.

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Ingo Molnar]

* Andrea Arcangeli <andrea@suse.de> wrote:

> As Ingo basically showed (and I agree), all current might_sleep seems
> suitable to be converted to cond_resched. I checked all them and
> there's none that seems to be called in a loop for no good reason. the
> ones in the semaphore are quite nice too since it's better to schedule
> right before taking the semaphore than after getting it. The one in
> the semaphore and in the copy-user are the only place where preempt
> seems to be lower overhead but in most other places a spinlock is
> being taken very near to the cond_resched.

yes. Btw., i'm not sure whether you've noticed but last week i've also
created a 'clean' variant of the patch. The latest version against -mm
is:

 http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-clean-2.6.7-mm7-H4

this one doesnt have any of the debugging/development helpers and
switches. I have still made it a .config option. Note how minimal the
patch became this way.

	Ingo

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Andrea Arcangeli]

Hi Ingo,

On Mon, Jul 19, 2004 at 01:59:52PM +0200, Ingo Molnar wrote:
> yes. Btw., i'm not sure whether you've noticed but last week i've also
> created a 'clean' variant of the patch. The latest version against -mm
> is:
> 
>  http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-clean-2.6.7-mm7-H4
> 
> this one doesnt have any of the debugging/development helpers and
> switches. I have still made it a .config option. Note how minimal the
> patch became this way.

as said in a part of a previous email might_sleep() just like BUG() can
be defined to noop.

cond_resched() is the API to use.

if you're scared that there are too many cond_resched (I'm not scared
and people should enable them anyways if they make a difference, they
still should be less than the number of spin_unlocks with preempt
enabled), then you should add a cond_resched_costly and add a config
option that turns it off. I think you don't even need to add the config
option, you can define cond_resched_costly as cond_resched, and to just
use it to mark the places that might be expensive.

Then you should change cond_resched to call might_sleep in the else
branch (as I discussed with Andrew last month).

this was the core of the patch I was playing with last month which
should be still valid and it solves the preprocessor issues with
cond_resched (and I hope the bug was not in the below code ;)

could you modify your patch accordingly? thanks!

Index: linux-2.5/include/linux/kernel.h
===================================================================
RCS file: /home/andrea/crypto/cvs/linux-2.5/include/linux/kernel.h,v
retrieving revision 1.53
diff -u -p -r1.53 kernel.h
--- linux-2.5/include/linux/kernel.h	27 Jun 2004 17:55:19 -0000	1.53
+++ linux-2.5/include/linux/kernel.h	13 Jul 2004 02:19:43 -0000
@@ -48,12 +48,23 @@ struct completion;
 #ifdef CONFIG_DEBUG_SPINLOCK_SLEEP
 void __might_sleep(char *file, int line);
 #define might_sleep() __might_sleep(__FILE__, __LINE__)
-#define might_sleep_if(cond) do { if (unlikely(cond)) might_sleep(); } while (0)
 #else
 #define might_sleep() do {} while(0)
-#define might_sleep_if(cond) do {} while (0)
 #endif
 
+#define need_resched() unlikely(test_thread_flag(TIF_NEED_RESCHED))
+
+extern void __cond_resched(void);
+#define cond_resched()				\
+do {						\
+	if (need_resched())			\
+		__cond_resched();		\
+	else					\
+		might_sleep();			\
+} while (0)
+
+#define cond_resched_if(cond) do { if (unlikely(cond)) cond_resched(); } while (0)
+
 #define abs(x) ({				\
 		int __x = (x);			\
 		(__x < 0) ? -__x : __x;		\
Index: linux-2.5/include/linux/sched.h
===================================================================
RCS file: /home/andrea/crypto/cvs/linux-2.5/include/linux/sched.h,v
retrieving revision 1.245
diff -u -p -r1.245 sched.h
--- linux-2.5/include/linux/sched.h	2 Jul 2004 17:31:23 -0000	1.245
+++ linux-2.5/include/linux/sched.h	13 Jul 2004 02:33:12 -0000
@@ -1012,18 +1013,6 @@ static inline int signal_pending(struct 
 {
 	return unlikely(test_tsk_thread_flag(p,TIF_SIGPENDING));
 }
-  
-static inline int need_resched(void)
-{
-	return unlikely(test_thread_flag(TIF_NEED_RESCHED));
-}
-
-extern void __cond_resched(void);
-static inline void cond_resched(void)
-{
-	if (need_resched())
-		__cond_resched();
-}
 
 /*
  * cond_resched_lock() - if a reschedule is pending, drop the given lock,

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Andrew Morton]

Andrea Arcangeli <andrea@suse.de> wrote:
>
>  if you're scared that there are too many cond_resched (I'm not scared
>  and people should enable them anyways if they make a difference, they
>  still should be less than the number of spin_unlocks with preempt
>  enabled), then you should add a cond_resched_costly and add a config
>  option that turns it off.

None of these approaches improves worst-case latency at all on SMP.  If
we're not going to address the SMP problem we could just make it UP-only,
in which case increased locking costs are a non-issue.

I'd prefer that we find a solution for SMP too though.

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Ingo Molnar]

* Andrew Morton <akpm@osdl.org> wrote:

> None of these approaches improves worst-case latency at all on SMP. 
> If we're not going to address the SMP problem we could just make it
> UP-only, in which case increased locking costs are a non-issue.
> 
> I'd prefer that we find a solution for SMP too though.

i have solved the fundamental SMP latency problems in the -Q7 patch, by
redesigning how SMP preemption is done. Here's the relevant changelog
entry:

[...]

the main change in this patch are more SMP latency fixes. The stock
kernel, even with CONFIG_PREEMPT enabled, didnt have any spin-nicely
preemption logic for the following, commonly used SMP locking
primitives: read_lock(), spin_lock_irqsave(), spin_lock_irq(),
spin_lock_bh(), read_lock_irqsave(), read_lock_irq(), read_lock_bh(),
write_lock_irqsave(), write_lock_irq(), write_lock_bh(). Only
spin_lock() and write_lock() [the two simplest cases] where covered.

In addition to the preemption latency problems, the _irq() variants in
the above list didnt do any IRQ-enabling while spinning - possibly
resulting in excessive irqs-off sections of code!

-Q7 fixes all of these latency problems: we now re-enable interrupts
while spinning in all possible cases, and a spinning op stays
preemptible if this is a beginning of a new critical section.

[...]

feedback from Mark H Johnson:

 http://lkml.org/lkml/2004/8/30/202

the latest patch is:

  http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.9-rc1-bk4-R0

i'm already in the process of cleaning up the patch and making it ready
for splitup & merge. The spinlock fixes will be amongst the first
patches i'll send you.

	Ingo

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

> i have solved the fundamental SMP latency problems in the -Q7 patch,
> by redesigning how SMP preemption is done. Here's the relevant
> changelog entry:

also this changelog explains the core changes that enable good
preemption latencies on SMP:

[...]

i took another look at SMP latencies, the last larger chunk of code that
produced millisec-category latencies. CONFIG_PREEMPT tries to solve some
of the SMP issues but there were still lots of problems remaining: main
problem area is spinlocks nested at multiple levels. If a piece of code
(e.g. the MM or ext3's journalling code) does the following:

        spin_lock(&spinlock_1);
        ...
        spin_lock(&spinlock_2);
        ...

then even with CONFIG_PREEMPT enabled, current kernels may spin on
spinlock_2 indefinitely. A number of critical sections break their long
paths by using cond_resched_lock(), but this does not break the path on
SMP, because need_resched() is not set in the above case.

(The -mm kernel introduced a couple of patches that try to drop
spinlocks unconditionally at a high frequency: but besides being a
kludge it's also a performance problem, we keep
dropping/waiting/retaking locks quite frequently. That solution also
doesnt solve the problem of cond_resched_lock() not working on SMP.)

to solve the problem i've introduced a new spinlock field,
lock->break_lock, which signals towards the holding CPU that a
spinlock-break is requested by another CPU. This field is only set if a
CPU is spinning in __preempt_spin_lock [at any locking depth], so the
default overhead is zero. I've extended cond_resched_lock() to check for
this flag - in this case we can also save a reschedule. I've added the
lock_need_resched(lock) and need_lockbreak(lock) methods to check for
the need to break out of a critical section.

preliminary results on a dual x86 box show a dramatic reduction in
latencies on SMP - where there used to be 5-10 msec latencies there are
close-to-UP latencies now. But it needs more testing.

[...]

	Ingo

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Takashi Iwai]

At Mon, 12 Jul 2004 20:01:51 -0400,
Paul Davis wrote:
> 
> >OK, thanks.  The problem areas there are the timer-based route cache
> >flushing and reiserfs.
> >
> >We can probably fix the route caceh thing by rescheduling the timer after
> >having handled 1000 routes or whatever, although I do wonder if this is a
> >thing we really need to bother about - what else was that machine up to?
> 
> i have one concern about this that i talked to takashi about when we
> were in bordeaux. it seems to me that the ALSA xrun debug stuff is
> measuring things when the interrupt handler for the ALSA device
> executes and detects an xrun. if the handler itself was delayed, then
> the stack trace for its execution doesn't or might not show what
> caused the delay. this means, perhaps, that we need to be rather
> careful interpreting these traces.

Well, it can catch up too long critical sections in most cases,
although it cannot measure the latency to wake up the audio thread.
So, the detection can't be 100% guaranteed.


Takashi

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Lee Revell]

On Mon, 2004-07-12 at 19:31, Andrew Morton wrote:
> Lee Revell <rlrevell@joe-job.com> wrote:
> >
> > On Sun, 2004-07-11 at 01:25, Andrew Morton wrote:
> > > What we need to do is to encourage audio testers to use ALSA drivers, to
> > > enable CONFIG_SND_DEBUG in the kernel build and to set
> > > /proc/asound/*/*/xrun_debug and to send us the traces which result from
> > > underruns.
> > > 
> > 
> > OK, here goes.  The following traces result from running JACK overnight
> > like so, on an otherwise idle system.  Hardware is a VIA EPIA 6000, with
> > a 600Mhz C3 processor.  Kernel is 2.6.7 + volunatary_preempt patch. 
> > voluntary_preempt and kernel_preemption are both on.
> > 
> > jackd -v --realtime -d alsa --outchannels 2 --rate 48000 --shorts
> > --playback --period 32  --nperiods 2
> > 
> > These settings require less than 666 microseconds scheduler latency. 
> > The average performance is quite good - 5-20 *microseconds*!
> 
> OK, thanks.  The problem areas there are the timer-based route cache
> flushing and reiserfs.
> 
> We can probably fix the route caceh thing by rescheduling the timer after
> having handled 1000 routes or whatever, although I do wonder if this is a
> thing we really need to bother about - what else was that machine up to?
> 

Gnutella client.  Forgot about that.  I agree, it is not reasonable to
expect low latency with this kind of network traffic happening.  I am
impressed it worked as well as it did.

> resierfs: yes, it's a problem.  I "fixed" it multiple times in 2.4, but the
> fixes ended up breaking the fs in subtle ways and I eventually gave up.
> 

Interesting.  There is an overwhelming consensus amongst Linux audio
folks that you should use reiserfs for low latency work.  Should I try
ext3?

Lee

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Andrew Morton]

Lee Revell <rlrevell@joe-job.com> wrote:
>
> > resierfs: yes, it's a problem.  I "fixed" it multiple times in 2.4, but the
>  > fixes ended up breaking the fs in subtle ways and I eventually gave up.
>  > 
> 
>  Interesting.  There is an overwhelming consensus amongst Linux audio
>  folks that you should use reiserfs for low latency work.

It seems to be misplaced.  A simple make-a-zillion-teeny-files test here
exhibits a 14 millisecond holdoff.

>  Should I try ext3?

ext3 is certainly better than that, but still has a couple of potential
problem spots.  ext2 is probably the best at this time.

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Lee Revell]

On Mon, 2004-07-12 at 20:46, Andrew Morton wrote:
> Lee Revell <rlrevell@joe-job.com> wrote:
> >
> > > resierfs: yes, it's a problem.  I "fixed" it multiple times in 2.4, but the
> >  > fixes ended up breaking the fs in subtle ways and I eventually gave up.
> >  > 
> > 
> >  Interesting.  There is an overwhelming consensus amongst Linux audio
> >  folks that you should use reiserfs for low latency work.
> 
> It seems to be misplaced.  A simple make-a-zillion-teeny-files test here
> exhibits a 14 millisecond holdoff.
> 
> >  Should I try ext3?
> 
> ext3 is certainly better than that, but still has a couple of potential
> problem spots.  ext2 is probably the best at this time.
> 

Here are a few more:

Jul 12 22:20:41 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 22:20:41 mindpipe kernel:  [__crc_totalram_pages+1387264/5353478] snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 22:20:41 mindpipe kernel:  [__crc_totalram_pages+1455070/5353478] snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 22:20:41 mindpipe kernel:  [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 22:20:41 mindpipe kernel:  [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 22:20:41 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 22:20:41 mindpipe kernel:  [unix_stream_recvmsg+135/1088] unix_stream_recvmsg+0x87/0x440
Jul 12 22:20:41 mindpipe kernel:  [avc_has_perm+72/96] avc_has_perm+0x48/0x60
Jul 12 22:20:41 mindpipe kernel:  [sock_aio_read+155/176] sock_aio_read+0x9b/0xb0
Jul 12 22:20:41 mindpipe kernel:  [do_sync_read+125/192] do_sync_read+0x7d/0xc0
Jul 12 22:20:41 mindpipe kernel:  [do_select+435/752] do_select+0x1b3/0x2f0
Jul 12 22:20:41 mindpipe kernel:  [vfs_read+223/256] vfs_read+0xdf/0x100
Jul 12 22:20:42 mindpipe kernel:  [sys_read+45/80] sys_read+0x2d/0x50
Jul 12 22:20:42 mindpipe kernel:  [syscall_call+7/11] syscall_call+0x7/0xb
Jul 12 22:20:42 mindpipe kernel:
Jul 12 22:20:44 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 22:20:44 mindpipe kernel:  [__crc_totalram_pages+1387264/5353478] snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 22:20:44 mindpipe kernel:  [__crc_totalram_pages+1455070/5353478] snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 22:20:44 mindpipe kernel:  [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 22:20:44 mindpipe kernel:  [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 22:20:44 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 22:20:44 mindpipe kernel:
Jul 12 22:20:49 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 22:20:49 mindpipe kernel:  [__crc_totalram_pages+1387264/5353478] snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 22:20:49 mindpipe kernel:  [__crc_totalram_pages+1455070/5353478] snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 22:20:49 mindpipe kernel:  [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 22:20:49 mindpipe kernel:  [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 22:20:49 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 22:20:49 mindpipe kernel:

Lee

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Andrew Morton]

Lee Revell <rlrevell@joe-job.com> wrote:
>
> Here are a few more:
> 
>  Jul 12 22:20:41 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
>  Jul 12 22:20:41 mindpipe kernel:  [__crc_totalram_pages+1387264/5353478] snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
>  Jul 12 22:20:41 mindpipe kernel:  [__crc_totalram_pages+1455070/5353478] snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
>  Jul 12 22:20:41 mindpipe kernel:  [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
>  Jul 12 22:20:41 mindpipe kernel:  [do_IRQ+155/352] do_IRQ+0x9b/0x160
>  Jul 12 22:20:41 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
>  Jul 12 22:20:41 mindpipe kernel:  [unix_stream_recvmsg+135/1088] unix_stream_recvmsg+0x87/0x440
>  Jul 12 22:20:41 mindpipe kernel:  [avc_has_perm+72/96] avc_has_perm+0x48/0x60
>  Jul 12 22:20:41 mindpipe kernel:  [sock_aio_read+155/176] sock_aio_read+0x9b/0xb0
>  Jul 12 22:20:41 mindpipe kernel:  [do_sync_read+125/192] do_sync_read+0x7d/0xc0
>  Jul 12 22:20:41 mindpipe kernel:  [do_select+435/752] do_select+0x1b3/0x2f0
>  Jul 12 22:20:41 mindpipe kernel:  [vfs_read+223/256] vfs_read+0xdf/0x100
>  Jul 12 22:20:42 mindpipe kernel:  [sys_read+45/80] sys_read+0x2d/0x50
>  Jul 12 22:20:42 mindpipe kernel:  [syscall_call+7/11] syscall_call+0x7/0xb
>  Jul 12 22:20:42 mindpipe kernel:
>  Jul 12 22:20:44 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
>  Jul 12 22:20:44 mindpipe kernel:  [__crc_totalram_pages+1387264/5353478] snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
>  Jul 12 22:20:44 mindpipe kernel:  [__crc_totalram_pages+1455070/5353478] snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
>  Jul 12 22:20:44 mindpipe kernel:  [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
>  Jul 12 22:20:44 mindpipe kernel:  [do_IRQ+155/352] do_IRQ+0x9b/0x160
>  Jul 12 22:20:44 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20

Oh dear, these don't make much sense.

You'll have the best chance of getting decent traces with
CONFIG_FRAME_POINTER=y and CONFIG_4KSTACKS=n, but I'm not sure that this
will help a lot.  Are you sure that CONFIG_PREEMPT is enabled on that
kernel?

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Lee Revell]

On Mon, 2004-07-12 at 23:59, Andrew Morton wrote:
> Lee Revell <rlrevell@joe-job.com> wrote:
> >
> > Here are a few more:
> > 
> >  Jul 12 22:20:41 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
> >  Jul 12 22:20:41 mindpipe kernel:  [__crc_totalram_pages+1387264/5353478] snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
> >  Jul 12 22:20:41 mindpipe kernel:  [__crc_totalram_pages+1455070/5353478] snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
> >  Jul 12 22:20:41 mindpipe kernel:  [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
> >  Jul 12 22:20:41 mindpipe kernel:  [do_IRQ+155/352] do_IRQ+0x9b/0x160
> >  Jul 12 22:20:41 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
> >  Jul 12 22:20:41 mindpipe kernel:  [unix_stream_recvmsg+135/1088] unix_stream_recvmsg+0x87/0x440
> >  Jul 12 22:20:41 mindpipe kernel:  [avc_has_perm+72/96] avc_has_perm+0x48/0x60
> >  Jul 12 22:20:41 mindpipe kernel:  [sock_aio_read+155/176] sock_aio_read+0x9b/0xb0
> >  Jul 12 22:20:41 mindpipe kernel:  [do_sync_read+125/192] do_sync_read+0x7d/0xc0
> >  Jul 12 22:20:41 mindpipe kernel:  [do_select+435/752] do_select+0x1b3/0x2f0
> >  Jul 12 22:20:41 mindpipe kernel:  [vfs_read+223/256] vfs_read+0xdf/0x100
> >  Jul 12 22:20:42 mindpipe kernel:  [sys_read+45/80] sys_read+0x2d/0x50
> >  Jul 12 22:20:42 mindpipe kernel:  [syscall_call+7/11] syscall_call+0x7/0xb
> >  Jul 12 22:20:42 mindpipe kernel:
> >  Jul 12 22:20:44 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
> >  Jul 12 22:20:44 mindpipe kernel:  [__crc_totalram_pages+1387264/5353478] snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
> >  Jul 12 22:20:44 mindpipe kernel:  [__crc_totalram_pages+1455070/5353478] snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
> >  Jul 12 22:20:44 mindpipe kernel:  [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
> >  Jul 12 22:20:44 mindpipe kernel:  [do_IRQ+155/352] do_IRQ+0x9b/0x160
> >  Jul 12 22:20:44 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
> 
> Oh dear, these don't make much sense.
> 
> You'll have the best chance of getting decent traces with
> CONFIG_FRAME_POINTER=y and CONFIG_4KSTACKS=n, but I'm not sure that this
> will help a lot.  Are you sure that CONFIG_PREEMPT is enabled on that
> kernel?

Yes:

root@mindpipe:/home/rlrevell/kernel-source/kernel-source-2.6.7# grep . /proc/sys/kernel/*_preemption
/proc/sys/kernel/kernel_preemption:1
/proc/sys/kernel/voluntary_preemption:1

My testing also uncovered a possible bug in the emu10k1 driver, I am
looking into this.  I don't think it's related because the log entries
are spaced minutes apart.  Could this be a driver problem?  Hardware?

Just as a reference point, what do you think is the longest delay I
*should* be seeing?  I recall hearing that BEOS guaranteed that
interrupts are never disabled for more than 50 usecs.  This seems
achievable, as the average delay I am seeing is 15 usecs.

I will set CONFIG_FRAME_POINTER=Y and CONFIG_4KSTACKS=n and test some
more.

Lee

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Ingo Molnar]

* Lee Revell <rlrevell@joe-job.com> wrote:

> Just as a reference point, what do you think is the longest delay I
> *should* be seeing?  I recall hearing that BEOS guaranteed that
> interrupts are never disabled for more than 50 usecs.  This seems
> achievable, as the average delay I am seeing is 15 usecs.

ATA hardirq latency can be as high as 700 usecs under load even on
modern hw, when big DMA requests are created with long scatter-gather
lists. We also moved some of the page IO completion code into irq
context which further increased hardirq latencies. Since these all touch
cold cachelines it all adds up quite quickly.

	Ingo

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

> ATA hardirq latency can be as high as 700 usecs under load even on
> modern hw, when big DMA requests are created with long scatter-gather
> lists. We also moved some of the page IO completion code into irq
> context which further increased hardirq latencies. Since these all
> touch cold cachelines it all adds up quite quickly.

typically all of this happens with irqs enabled (unmask=1), but it still
increases scheduling latencies.

with the default DMA setup of today's ATA hw there can be a maximum of
256 entries in the sg-table all zapped in ide_end_request() ->
__end_that_request_first().

Plus the IDE driver also builds the sg-table of the next request in
hardirq context. (ide_build_dmatable() and ide_build_sglist()).

	Ingo

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Lee Revell]

On Mon, 2004-07-19 at 06:48, Ingo Molnar wrote:
> * Lee Revell <rlrevell@joe-job.com> wrote:
> 
> > Just as a reference point, what do you think is the longest delay I
> > *should* be seeing?  I recall hearing that BEOS guaranteed that
> > interrupts are never disabled for more than 50 usecs.  This seems
> > achievable, as the average delay I am seeing is 15 usecs.
> 
> ATA hardirq latency can be as high as 700 usecs under load even on
> modern hw, when big DMA requests are created with long scatter-gather
> lists. We also moved some of the page IO completion code into irq
> context which further increased hardirq latencies. Since these all touch
> cold cachelines it all adds up quite quickly.
> 

Does this scale in a linear fashion with respect to CPU speed?  You
mentioned you were testing on a 2Ghz machine, does 700 usecs on that
translate to 2800 usecs on a 500Mhz box?

On a 2Ghz machine, 700 usecs is about one million CPU cycles.  In other
words, the highest priority process can become runnable, then have to
wait *one million cycles* to get the CPU.

How much I/O do you allow to be in flight at once?  It seems like by
decreasing the maximum size of I/O that you handle in one interrupt you
could improve this quite a bit.  Disk throughput is good enough, anyone
in the real world who would feel a 10% hit would just throw hardware at
the problem.

Lee

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Ingo Molnar]

* Lee Revell <rlrevell@joe-job.com> wrote:

> Does this scale in a linear fashion with respect to CPU speed?  You
> mentioned you were testing on a 2Ghz machine, does 700 usecs on that
> translate to 2800 usecs on a 500Mhz box?

given that this particular latency is dominated by cachemisses the DRAM
latency controls it too which is independent of CPU MHz. Wrt. cachemiss
costs, newer CPUs typically have twice the L2-cache line size (so it
takes more bus cycles to fetch it) - the improvements in bandwidth of
fetching a single line should offset most of this. DRAM latencies didnt
improve much in the past 10 years so that's almost a constant between a
500MHz/100MHz(system-bus) vs. 2GHz/400MHz system. So i'd guesstimate a
500 MHz box to do somewhere around 1000-1500 usecs.

> How much I/O do you allow to be in flight at once?  It seems like by
> decreasing the maximum size of I/O that you handle in one interrupt
> you could improve this quite a bit.  Disk throughput is good enough,
> anyone in the real world who would feel a 10% hit would just throw
> hardware at the problem.

i'm not sure whether this particular value (max # of sg-entries per IO
op) is runtime tunable. Jens? Might make sense to enable elvtune-alike
tunability of this value.

limiting the in-flight IO will only work with IDE/PATA that doesnt have
multiple commands in flight for a given disk. SATA and SCSI handles
multiple command completions per IRQ invocation so limiting the size of
a single IO op has less effect there.

	Ingo

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Lee Revell]

On Tue, 2004-07-20 at 02:12, Ingo Molnar wrote:
> * Lee Revell <rlrevell@joe-job.com> wrote:
> 
> > Does this scale in a linear fashion with respect to CPU speed?  You
> > mentioned you were testing on a 2Ghz machine, does 700 usecs on that
> > translate to 2800 usecs on a 500Mhz box?
> 
> given that this particular latency is dominated by cachemisses the DRAM
> latency controls it too which is independent of CPU MHz. Wrt. cachemiss
> costs, newer CPUs typically have twice the L2-cache line size (so it
> takes more bus cycles to fetch it) - the improvements in bandwidth of
> fetching a single line should offset most of this. DRAM latencies didnt
> improve much in the past 10 years so that's almost a constant between a
> 500MHz/100MHz(system-bus) vs. 2GHz/400MHz system. So i'd guesstimate a
> 500 MHz box to do somewhere around 1000-1500 usecs.
> 

The particular system I am working with is a Via EPIA M-6000, which is
quite new but has a 600Mhz CPU due to it being fanless.  I would imagine
this would put it closer to a 'modern system' in this regard, as it uses
DDR266.

> > How much I/O do you allow to be in flight at once?  It seems like by
> > decreasing the maximum size of I/O that you handle in one interrupt
> > you could improve this quite a bit.  Disk throughput is good enough,
> > anyone in the real world who would feel a 10% hit would just throw
> > hardware at the problem.
> 
> i'm not sure whether this particular value (max # of sg-entries per IO
> op) is runtime tunable. Jens? Might make sense to enable elvtune-alike
> tunability of this value.

Yes, this would be a nice improvement.

> 
> limiting the in-flight IO will only work with IDE/PATA that doesnt have
> multiple commands in flight for a given disk. SATA and SCSI handles
> multiple command completions per IRQ invocation so limiting the size of
> a single IO op has less effect there.
> 

But the current behavior only causes latency problems for an IDE system,
so if this were made runtime-tunable then it would only be an issue for
SATA, right?  This would cover 99.9% of audio users, who would gladly
trade some disk throughput for lower latency.  You can record a *lot* of
tracks with even a few MB/s of disk throughput.

Lee

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Ingo Molnar]

* Lee Revell <rlrevell@joe-job.com> wrote:

> But the current behavior only causes latency problems for an IDE
> system, so if this were made runtime-tunable then it would only be an
> issue for SATA, right?  This would cover 99.9% of audio users, who
> would gladly trade some disk throughput for lower latency.  You can
> record a *lot* of tracks with even a few MB/s of disk throughput.

it's an issue for all block IO drivers that do IO completions from IRQ
context and that can do DMA - i.e. every block IO hardware that uses
interrupts. This includes SCSI too. In fact for SCSI it's a norm to have
tagged queueing active so there the latencies ought to be even higher
(although i havent measured this). IDE/PATA's limitation in this regard
limits latencies as well.

being able to control the max size of sg-tables and the max # of
outstanding commands per IRQ source [this later should already be
possible via driver options] should enable us to control the maximum
hardirq latency introduced by block IO.

(if the hardware & disk is fast enough, or you use a high # of
controllers and disks then you could still overload your system with a
stream of interrupts and cause unbound scheduling latencies - but this
is a separate problem.)

	Ingo

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Lee Revell]

On Tue, 2004-07-20 at 14:11, Ralf Beck wrote:
> > it's an issue for all block IO drivers that do IO completions from IRQ
> > context and that can do DMA - i.e. every block IO hardware that uses
> > interrupts. This includes SCSI too. In fact for SCSI it's a norm to have
> 
> I renew a question i asked earlier.
> 
> To my understanding, on a SMP or hyperthreading system, disabling of
> IRQs is always local to one (virtual on HT) cpu.
> 
> So would it be possible to get ultralow latency by simply hardlock all irqs 
> and processes to cpu1 and the irq triggering the audiothread (together with 
> the audiothread) to cpu 2 using the sched_affinity and irq-affinity 
> capabilites of the kernel?
> 
> This would be an easy to use lowlatency hardware patch for  linux audio users
> with SMP/HT systems. Anybody knows?
> 
> I'm currently thinking about getting a new system and consider a dualsystem if 
> this worked.

Should work.  For example, the RTLinux people report excellent results
on SMP systems by binding all RT threads to one CPU and having the Linux
part of the system run on the other.  This is just a "softer" version of
that setup.  Even if there are cases where IRQs are disabled globally,
it would be an improvement.  I suspect you are not getting much of a
response because no one has actually tested it with an audio system.

Lee

Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Benno Senoner]

While locking a RT process to a CPU to achieve even lower latencies 
might be useful to some
the general userbase wants good latencies on simple UP, non HT-enabled 
hardware too.
(AMD is gaining marketshare and we cannot simply expect that good 
multimedia performance (aka low latency)
can be achieved only on those SMP/HT boxes where the truth is that 
except in the case of really crappy hardware,
those common UP  machines are actually capable to delives  latencies in 
the range of dozen of usecs.
(taking the RTLinux benchmarks as reference).
Of course Linux != RTLinux and we never expect nor pretend Linux to 
match the response time of RTLinux but
as said earlier latencies at around 1msec should are doable on decent UP 
boxes and this is more than
adequate to run demanding real time audio applications like software 
synthesizers, samples, real time effects etc.
The only hope is that the hard work from Ingo, Andrew and all the others 
gets soon integrated in the mainstream
2.6 kernel so all Linux users can take run demanding multimedia 
applications out of the box without going through
the painful patching,kernel recompiling etc.
Or is this still not realistic at this point ?
 (see the old issue with kernel 2.4,  low latency patches were too 
unclean to make it into the
ufficial kernel tree).

Plus what's very important is that every kernel developer and driver 
developer (even thirdparty, especially those
that do closed source stuff like Nvidia etc) takes into account the 
latency problems that code paths that run for
too long time (or disable IRQs for too long etc) might create.
While I'm not a kernel expert I assume the premptible kernel alleviates 
this problem but I guess it still cannot
completely get rid of low latency-unfriendly routines and drivers.
Perhaps in upcoming Linux kernel/driver programming books this issue 
should be stressed out and
mentioned prominently in order to avoid the eternal:
fixing latencies in kernel x.y.z -> new kernel versions with new 
drivers/changes/enhancements comes out
-> latencies are bad again -> fix latencies in this new kernel version  
.....  ( iterate at infinitum)

cheers,
Benno
http://www.linuxsampler.org


Lee Revell wrote:

>On Tue, 2004-07-20 at 14:11, Ralf Beck wrote:
>  
>
>>>it's an issue for all block IO drivers that do IO completions from IRQ
>>>context and that can do DMA - i.e. every block IO hardware that uses
>>>interrupts. This includes SCSI too. In fact for SCSI it's a norm to have
>>>      
>>>
>>I renew a question i asked earlier.
>>
>>To my understanding, on a SMP or hyperthreading system, disabling of
>>IRQs is always local to one (virtual on HT) cpu.
>>
>>So would it be possible to get ultralow latency by simply hardlock all irqs 
>>and processes to cpu1 and the irq triggering the audiothread (together with 
>>the audiothread) to cpu 2 using the sched_affinity and irq-affinity 
>>capabilites of the kernel?
>>
>>This would be an easy to use lowlatency hardware patch for  linux audio users
>>with SMP/HT systems. Anybody knows?
>>
>>I'm currently thinking about getting a new system and consider a dualsystem if 
>>this worked.
>>    
>>
>
>Should work.  For example, the RTLinux people report excellent results
>on SMP systems by binding all RT threads to one CPU and having the Linux
>part of the system run on the other.  This is just a "softer" version of
>that setup.  Even if there are cases where IRQs are disabled globally,
>it would be an improvement.  I suspect you are not getting much of a
>response because no one has actually tested it with an audio system.
>
>Lee
>
>
>  
>

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Lee Revell]

On Tue, 2004-07-20 at 19:49, Benno Senoner wrote:
> While locking a RT process to a CPU to achieve even lower latencies 
> might be useful to some
> the general userbase wants good latencies on simple UP, non HT-enabled 
> hardware too.
> (AMD is gaining marketshare and we cannot simply expect that good 
> multimedia performance (aka low latency)
> can be achieved only on those SMP/HT boxes where the truth is that 
> except in the case of really crappy hardware,
> those common UP  machines are actually capable to delives  latencies in 
> the range of dozen of usecs.
> (taking the RTLinux benchmarks as reference).
> Of course Linux != RTLinux and we never expect nor pretend Linux to 
> match the response time of RTLinux but
> as said earlier latencies at around 1msec should are doable on decent UP 
> boxes and this is more than
> adequate to run demanding real time audio applications like software 
> synthesizers, samples, real time effects etc.
> The only hope is that the hard work from Ingo, Andrew and all the others 
> gets soon integrated in the mainstream
> 2.6 kernel so all Linux users can take run demanding multimedia 
> applications out of the box without going through
> the painful patching,kernel recompiling etc.
> Or is this still not realistic at this point ?
>  (see the old issue with kernel 2.4,  low latency patches were too 
> unclean to make it into the
> ufficial kernel tree).
> 

There are still a few areas that need work, ioctl gives me problems, but
the latest 2.6 kernels are quite good.  If you look at the 'clean'
version of the voluntary kernel preemption patch it is pretty small.  My
understanding is that the kernel is already preemptible anytime that a
spin lock (including the BKL) is not held, and that the voluntary kernel
preemption patch adds some scheduling points in places where it is safe
to sleep, but preemption is disabled because we are holding the BKL, and
that the number of these should approach zero as the kernel is improved
anyway.


> Plus what's very important is that every kernel developer and driver 
> developer (even thirdparty, especially those
> that do closed source stuff like Nvidia etc) takes into account the 
> latency problems that code paths that run for
> too long time (or disable IRQs for too long etc) might create.
> While I'm not a kernel expert I assume the premptible kernel alleviates 
> this problem but I guess it still cannot
> completely get rid of low latency-unfriendly routines and drivers.

Yes, this is important.  One problem I had recently with the Via EPIA
board was that unless 2D acceleration was disabled by setting 'Option
"NoAccel"' in /etc/X11/XF86Config-4, overloading the X server would
cause interrupts from the soundcard to be completely disabled for tens
of milliseconds.  Users should keep in mind that by using 2D or 3D
hardware acceleration in X, you are allowing the X server to directly
access hardware, which can have very bad results if the driver is
buggy.  I am not sure the kernel can do anything about this.

Lee

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Timothy Miller]

Lee Revell wrote:

> There are still a few areas that need work, ioctl gives me problems, but
> the latest 2.6 kernels are quite good.  If you look at the 'clean'
> version of the voluntary kernel preemption patch it is pretty small.  My
> understanding is that the kernel is already preemptible anytime that a
> spin lock (including the BKL) is not held, and that the voluntary kernel
> preemption patch adds some scheduling points in places where it is safe
> to sleep, but preemption is disabled because we are holding the BKL, and
> that the number of these should approach zero as the kernel is improved
> anyway.


That's confusing to me.  It was my understanding that the BKL is used to 
completely lock down the kernel so that no other CPU can have a process 
get into the kernel... something like how SMP was done under 2.0.

So, if you sleep during a BKL, wouldn't that imply that nothing else 
would be allowed to enter the kernel until after the kernel thread that 
took the lock wakes up and releases the lock?

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Florian Schmidt]

On Tue, 20 Jul 2004 20:32:37 -0400
Lee Revell <rlrevell@joe-job.com> wrote:

> 
> Yes, this is important.  One problem I had recently with the Via EPIA
> board was that unless 2D acceleration was disabled by setting 'Option
> "NoAccel"' in /etc/X11/XF86Config-4, overloading the X server would
> cause interrupts from the soundcard to be completely disabled for tens
> of milliseconds.  Users should keep in mind that by using 2D or 3D
> hardware acceleration in X, you are allowing the X server to directly
> access hardware, which can have very bad results if the driver is
> buggy.  I am not sure the kernel can do anything about this.

Hi,

interesting that you mention the Xserver. I use a dual graphics card setup atm [Nvidia GF3 TI and some matrox pci card]. The nvidia card seems to work flawlessly even with HW accelleration [i use nvidias evil binary only drivers]. The matrox card OTH disturbs the soundcard severely. Whenever i have activity on my second monitor i get sound artefacts in jack's output [no cracklling, it's rather as if the volume is set to 0 for short moments and then back to normal]. There's a certain chance that this artefact produces an xrun. I suppose it's because the card is on the pci bus.

I figured it's maybe an irq issue problem, but whatever slot i put the gfx card in - it made no difference [btw: how do i find out which resources this card uses? it is not shown by /proc/interrupts]. I also tried putting the soundcard in many different slots to maybe get it on higher prio irq, but it always gets irq 5 [according to /proc/interrupts]..

Should i try a different 2nd gfx card? Should i avoid pci gfx cards at all costs? Will i just have to live w/o second monitor?  How do i find out which hw resources X is really using?

Florian Schmidt
-- 
Palimm Palimm!
Sounds/Ware:
http://affenbande.org/~tapas/

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Thomas Charbonnel]

Florian Schmidt wrote :
> Hi,
> 
> interesting that you mention the Xserver. I use a dual graphics card setup atm [Nvidia GF3 TI and some matrox pci card]. The nvidia card seems to work flawlessly even with HW accelleration [i use nvidias evil binary only drivers]. The matrox card OTH disturbs the soundcard severely. Whenever i have activity on my second monitor i get sound artefacts in jack's output [no cracklling, it's rather as if the volume is set to 0 for short moments and then back to normal]. There's a certain chance that this artefact produces an xrun. I suppose it's because the card is on the pci bus.
> 
> I figured it's maybe an irq issue problem, but whatever slot i put the gfx card in - it made no difference [btw: how do i find out which resources this card uses? it is not shown by /proc/interrupts]. I also tried putting the soundcard in many different slots to maybe get it on higher prio irq, but it always gets irq 5 [according to /proc/interrupts]..
> 
> Should i try a different 2nd gfx card? Should i avoid pci gfx cards at all costs? Will i just have to live w/o second monitor?  How do i find out which hw resources X is really using?
> 
> Florian Schmidt

You could try to adjust the pci latency timer value of your graphic card
and sound card, see this link for a paper on the subject by Daniel
Robbins :
http://www-106.ibm.com/developerworks/library/l-hw2.html

Thomas

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Fernando Pablo Lopez-Lezcano]

On Wed, 2004-07-21 at 03:53, Florian Schmidt wrote:
> On Tue, 20 Jul 2004 20:32:37 -0400
> Lee Revell <rlrevell@joe-job.com> wrote:
> > Yes, this is important.  One problem I had recently with the Via EPIA
> > board was that unless 2D acceleration was disabled by setting 'Option
> > "NoAccel"' in /etc/X11/XF86Config-4, overloading the X server would
> > cause interrupts from the soundcard to be completely disabled for tens
> > of milliseconds.  Users should keep in mind that by using 2D or 3D
> > hardware acceleration in X, you are allowing the X server to directly
> > access hardware, which can have very bad results if the driver is
> > buggy.  I am not sure the kernel can do anything about this.
> 
> Hi,
> 
> interesting that you mention the Xserver. I use a dual graphics card setup atm 
> [Nvidia GF3 TI and some matrox pci card]. The nvidia card seems to work flawlessly 
> even with HW accelleration [i use nvidias evil binary only drivers]. The matrox 
> card OTH disturbs the soundcard severely. Whenever i have activity on my second 
> monitor i get sound artefacts in jack's output [no cracklling, it's rather as 
> if the volume is set to 0 for short moments and then back to normal]. There's 
> a certain chance that this artefact produces an xrun. I suppose it's because 
> the card is on the pci bus.

The mga dri kernel driver shares a problem with the radeon (which I use
a lot) and the r128 drivers. They have high latency points that reach
10-15 msecs in normal operation[*]. I have a very old patch (not mine, I
don't quite remember where I got it from) that solves this, but it is
not a "legal" patch in the sense that in schedules with a lock held. It
seems to work but it will lock the computer at some point. AFAIK there
is no proper patch for this latency point at this time. Turning off
acceleration should get rid of the latency spikes with the usual
tradeoff of slow video performance. 

-- Fernando

[*] with the (bad) patch:
http://ccrma.stanford.edu/~nando/latencytest/20040323/2.6.4-1.279.ll.ccrma.radeon/
without the patch:
http://ccrma.stanford.edu/~nando/latencytest/20040323/2.6.4-1.279.ll.ccrma/

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Andrew McGregor]

--On Wednesday, 21 July 2004 12:53 p.m. +0200 Florian Schmidt 
<mista.tapas@gmx.net> wrote:

> Hi,
>
> interesting that you mention the Xserver. I use a dual graphics card
> setup atm [Nvidia GF3 TI and some matrox pci card]. The nvidia card seems
> to work flawlessly even with HW accelleration [i use nvidias evil binary
> only drivers]. The matrox card OTH disturbs the soundcard severely.
> Whenever i have activity on my second monitor i get sound artefacts in
> jack's output [no cracklling, it's rather as if the volume is set to 0
> for short moments and then back to normal]. There's a certain chance that
> this artefact produces an xrun. I suppose it's because the card is on the
> pci bus.

<snip>

> Should i try a different 2nd gfx card? Should i avoid pci gfx cards at
> all costs? Will i just have to live w/o second monitor?  How do i find
> out which hw resources X is really using?
>
> Florian Schmidt


It is a PCI bus issue.  You simply don't have enough PCI bus cycles 
available to do what you want to do.  The resource you're running out of is 
bus bandwidth, and there's nothing to be done about it, other than remove 
the PCI gfx card from the system.

If you get another dualhead AGP graphics card (anything will do), the 
problem should go away.  We have a developer who does lowlatency 
multichannel sound stuff on a machine with a Matrox G450 dualhead card no 
problem.  I expect my own system (Radeon 9800 Pro and M-Audio 1010LT audio) 
would be fine dualhead too, although I only run it singlehead at the 
moment.  The 1010LT is 10 channels in and out of 24-bit 96kHz audio and 
works great down to 1.5ms buffers, so it is no small bus load itself.

Andrew


---------
Andrew McGregor
Director, Scientific Advisor
IndraNet Technologies Ltd
http://www.indranet-technologies.com/

-----BEGIN GEEK CODE BLOCK-----
Version: 3.1
GS/E/B/PA/SS d+(++) s+:+ a C++$ ULS++++ !P+++(---)$ L++++$ E++ W++ !N
w(+++) !O() M++ V--() Y+$ PGP+ t- !5? X- !R !tv@ b++(++++) DI++ D+++@ G
e+++ h(*)@ r%
------END GEEK CODE BLOCK------

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Florian Schmidt]

On Fri, 23 Jul 2004 10:25:07 +1200
Andrew McGregor <andrew@indranet.co.nz> wrote:

> It is a PCI bus issue.  You simply don't have enough PCI bus cycles 
> available to do what you want to do.  The resource you're running out of is 
> bus bandwidth, and there's nothing to be done about it, other than remove 
> the PCI gfx card from the system.
> 
> If you get another dualhead AGP graphics card (anything will do), the 
> problem should go away.  We have a developer who does lowlatency 
> multichannel sound stuff on a machine with a Matrox G450 dualhead card no 
> problem.  I expect my own system (Radeon 9800 Pro and M-Audio 1010LT audio) 
> would be fine dualhead too, although I only run it singlehead at the 
> moment.  The 1010LT is 10 channels in and out of 24-bit 96kHz audio and 
> works great down to 1.5ms buffers, so it is no small bus load itself.

Hi,

thanks for the advice. I have tried some more pci cards and a very slow Virge 64 or something actually reduced the problem. Not completely though. I think i'll get me one of those G450 dual head cards..  

Florian Schmidt

-- 
Palimm Palimm!
http://affenbande.org/~tapas/

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Benno Senoner]

Lee Revell wrote:

>anyway.
>
>
>  
>
>>Plus what's very important is that every kernel developer and driver 
>>developer (even thirdparty, especially those
>>that do closed source stuff like Nvidia etc) takes into account the 
>>latency problems that code paths that run for
>>too long time (or disable IRQs for too long etc) might create.
>>While I'm not a kernel expert I assume the premptible kernel alleviates 
>>this problem but I guess it still cannot
>>completely get rid of low latency-unfriendly routines and drivers.
>>    
>>
>
>Yes, this is important.  One problem I had recently with the Via EPIA
>board was that unless 2D acceleration was disabled by setting 'Option
>"NoAccel"' in /etc/X11/XF86Config-4, overloading the X server would
>cause interrupts from the soundcard to be completely disabled for tens
>of milliseconds.  Users should keep in mind that by using 2D or 3D
>hardware acceleration in X, you are allowing the X server to directly
>access hardware, which can have very bad results if the driver is
>buggy.  I am not sure the kernel can do anything about this.
>  
>

that's bad news.
VIA markets those mini-itx mainboards (with onboard audio/video/network) 
as multimedia appliances and
therefore I'd expect the hardware providing low latencies  when both 
video acceleration and audio is used.
Hopefully only a driver issue (as in most of cases)

Since VIA released the unichrome (the gfx chipset contained in their 
mainboards) sources someone should
contact these folks :
http://unichrome.sourceforge.net/
to check what is causing those latency spikes ?
Any unichrome developer lurking on LKML ?

cheers,
Benno
http://www.linuxsampler.org

>Lee
>
>
>  
>

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Jens Axboe]

On Tue, Jul 20 2004, Ingo Molnar wrote:
> > How much I/O do you allow to be in flight at once?  It seems like by
> > decreasing the maximum size of I/O that you handle in one interrupt
> > you could improve this quite a bit.  Disk throughput is good enough,
> > anyone in the real world who would feel a 10% hit would just throw
> > hardware at the problem.
> 
> i'm not sure whether this particular value (max # of sg-entries per IO
> op) is runtime tunable. Jens? Might make sense to enable elvtune-alike
> tunability of this value.

elvtune is long dead :-)

it's not tweakable right now, but if you wish to experiment you just
need to add a line to ide-disk.c:idedisk_setup() - pseudo patch:

+	blk_queue_max_sectors(drive->queue, 32);
+
	printk("%s: max request size: %dKiB\n", drive->name, drive->queue->max_sectors / 2);

	/* Extract geometry if we did not already have one for the drive */

above will limit max request to 16kb, experiment as you see fit.

-- 
Jens Axboe

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Lee Revell]

On Tue, 2004-07-20 at 08:19, Jens Axboe wrote:
> On Tue, Jul 20 2004, Ingo Molnar wrote:
> > > How much I/O do you allow to be in flight at once?  It seems like by
> > > decreasing the maximum size of I/O that you handle in one interrupt
> > > you could improve this quite a bit.  Disk throughput is good enough,
> > > anyone in the real world who would feel a 10% hit would just throw
> > > hardware at the problem.
> > 
> > i'm not sure whether this particular value (max # of sg-entries per IO
> > op) is runtime tunable. Jens? Might make sense to enable elvtune-alike
> > tunability of this value.
> 
> elvtune is long dead :-)
> 
> it's not tweakable right now, but if you wish to experiment you just
> need to add a line to ide-disk.c:idedisk_setup() - pseudo patch:
> 
> +	blk_queue_max_sectors(drive->queue, 32);
> +
> 	printk("%s: max request size: %dKiB\n", drive->name, drive->queue->max_sectors / 2);
> 
> 	/* Extract geometry if we did not already have one for the drive */
> 
> above will limit max request to 16kb, experiment as you see fit.

I will give this a try.

Using 2.6.8-rc1-mm1, with La Monte H. P. Yarroll's patch to daemonize
softirqs, running two jackds (one in the playback direction, one in
capture, to work around an issue with the emu10k1 driver) with 2 periods
of 32 frames (666 usec), I am no longer seeing any XRUNS at all once the
jackd processes are up and running.  I tried stressing the filesystem by
recompiling ALSA while running a 'du /' and a 'find / -ls'.  The
scheduling jitter peaks occasionally at around 100000 CPU cycles (166
usecs) every few seconds, but this is as high as it goes.

This is as good or better than a 2.4 kernel with the low latency
patches.

I do not have voluntary preemption enabled, as I do not have a version
of the patch that corresponds to this kernel.

The only XRUNS I still get seem related to ioctl() and mlockall().  
These both cause an XRUN in one jackd process when starting another:

Jul 20 16:25:46 mindpipe kernel: ALSA /home/rlrevell/cvs/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 20 16:25:46 mindpipe kernel:  [dump_stack+23/32] dump_stack+0x17/0x20
Jul 20 16:25:46 mindpipe kernel:  [__crc_totalram_pages+115469/3518512] snd_pcm_period_elapsed+0x2c7/0x400 [snd_pcm]
Jul 20 16:25:46 mindpipe kernel:  [__crc_totalram_pages+291773/3518512] snd_emu10k1_interrupt+0x337/0x3c0 [snd_emu10k1]
Jul 20 16:25:46 mindpipe kernel:  [handle_IRQ_event+51/96] handle_IRQ_event+0x33/0x60
Jul 20 16:25:46 mindpipe kernel:  [do_IRQ+161/320] do_IRQ+0xa1/0x140
Jul 20 16:25:46 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 20 16:25:46 mindpipe kernel:  [do_anonymous_page+124/384] do_anonymous_page+0x7c/0x180
Jul 20 16:25:46 mindpipe kernel:  [do_no_page+78/784] do_no_page+0x4e/0x310
Jul 20 16:25:46 mindpipe kernel:  [handle_mm_fault+193/368] handle_mm_fault+0xc1/0x170
Jul 20 16:25:46 mindpipe kernel:  [get_user_pages+337/960] get_user_pages+0x151/0x3c0
Jul 20 16:25:46 mindpipe kernel:  [make_pages_present+104/144] make_pages_present+0x68/0x90
Jul 20 16:25:46 mindpipe kernel:  [mlock_fixup+141/176] mlock_fixup+0x8d/0xb0
Jul 20 16:25:46 mindpipe kernel:  [do_mlockall+112/144] do_mlockall+0x70/0x90
Jul 20 16:25:46 mindpipe kernel:  [sys_mlockall+150/160] sys_mlockall+0x96/0xa0
Jul 20 16:25:46 mindpipe kernel:  [syscall_call+7/11] syscall_call+0x7/0xb
Jul 20 16:25:46 mindpipe kernel: IRQ: delay =  9213566 cycles, jitter = 8820554

Jul 20 16:54:02 mindpipe kernel: ALSA /home/rlrevell/cvs/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D2c
Jul 20 16:54:02 mindpipe kernel:  [dump_stack+23/32] dump_stack+0x17/0x20
Jul 20 16:54:02 mindpipe kernel:  [__crc_totalram_pages+115469/3518512] snd_pcm_period_elapsed+0x2c7/0x400 [snd_pcm]
Jul 20 16:54:02 mindpipe kernel:  [__crc_totalram_pages+291159/3518512] snd_emu10k1_interrupt+0xd1/0x3c0 [snd_emu10k1]
Jul 20 16:54:02 mindpipe kernel:  [handle_IRQ_event+51/96] handle_IRQ_event+0x33/0x60
Jul 20 16:54:02 mindpipe kernel:  [do_IRQ+161/320] do_IRQ+0xa1/0x140
Jul 20 16:54:02 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 20 16:54:02 mindpipe kernel:  [__crc_totalram_pages+86289/3518512] snd_pcm_hw_rule_div+0x4b/0x60 [snd_pcm]
Jul 20 16:54:02 mindpipe kernel:  [__crc_totalram_pages+77959/3518512] snd_pcm_hw_refine+0x371/0x4a0 [snd_pcm]
Jul 20 16:54:02 mindpipe kernel:  [__crc_totalram_pages+111312/3518512] snd_pcm_hw_param_mask+0x3a/0x50 [snd_pcm]
Jul 20 16:54:02 mindpipe kernel:  [__crc_totalram_pages+882570/3518512] snd_pcm_oss_change_params+0xf4/0x850 [snd_pcm_oss]
Jul 20 16:54:02 mindpipe kernel:  [__crc_totalram_pages+884572/3518512] snd_pcm_oss_get_active_substream+0x76/0x90 [snd_pcm_oss]
Jul 20 16:54:02 mindpipe kernel:  [__crc_totalram_pages+888499/3518512] snd_pcm_oss_get_formats+0x1d/0x120 [snd_pcm_oss]
Jul 20 16:54:02 mindpipe kernel:  [__crc_totalram_pages+888796/3518512] snd_pcm_oss_set_format+0x26/0x60 [snd_pcm_oss]
Jul 20 16:54:02 mindpipe kernel:  [__crc_totalram_pages+894737/3518512] snd_pcm_oss_ioctl+0x49b/0x740 [snd_pcm_oss]
Jul 20 16:54:02 mindpipe kernel:  [sys_ioctl+256/608] sys_ioctl+0x100/0x260
Jul 20 16:54:02 mindpipe kernel:  [syscall_call+7/11] syscall_call+0x7/0xb
Jul 20 16:54:02 mindpipe kernel: IRQ: delay =  6702030 cycles, jitter = 6302828

Lee

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Lee Revell]

On Tue, 2004-07-20 at 08:19, Jens Axboe wrote:
> On Tue, Jul 20 2004, Ingo Molnar wrote:
> > > How much I/O do you allow to be in flight at once?  It seems like by
> > > decreasing the maximum size of I/O that you handle in one interrupt
> > > you could improve this quite a bit.  Disk throughput is good enough,
> > > anyone in the real world who would feel a 10% hit would just throw
> > > hardware at the problem.
> > 
> > i'm not sure whether this particular value (max # of sg-entries per IO
> > op) is runtime tunable. Jens? Might make sense to enable elvtune-alike
> > tunability of this value.
> 
> elvtune is long dead :-)
> 
> it's not tweakable right now, but if you wish to experiment you just
> need to add a line to ide-disk.c:idedisk_setup() - pseudo patch:
> 
> +	blk_queue_max_sectors(drive->queue, 32);
> +
> 	printk("%s: max request size: %dKiB\n", drive->name, drive->queue->max_sectors / 2);
> 
> 	/* Extract geometry if we did not already have one for the drive */
> 
> above will limit max request to 16kb, experiment as you see fit.

I tested this and the improvement is drastic.  With the default value of
1024KB, running 'bonnie' produced XRUNS of 10+ ms.  When this is changed
to 16KB, running bonnie only produces latency spikes of up to about 90
usecs, I did not see a single one hit 100usecs.

According to hdparm, the throughput is still quite good (42MB/sec on a 
sub-$100 IDE drive).

This should definitely be made tunable, I would imagine this would be
easy to put in /proc.  The default could stay at 1024KB, and users with
low latency requirements could lower it.

I am currently testing the effect on throughput and will have some
better numbers soon.

Lee

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Lee Revell]

On Sat, 2004-07-24 at 00:07, Lee Revell wrote:
> On Tue, 2004-07-20 at 08:19, Jens Axboe wrote:
> > On Tue, Jul 20 2004, Ingo Molnar wrote:
> > > > How much I/O do you allow to be in flight at once?  It seems like by
> > > > decreasing the maximum size of I/O that you handle in one interrupt
> > > > you could improve this quite a bit.  Disk throughput is good enough,
> > > > anyone in the real world who would feel a 10% hit would just throw
> > > > hardware at the problem.
> > > 
> > it's not tweakable right now, but if you wish to experiment you just
> > need to add a line to ide-disk.c:idedisk_setup() - pseudo patch:
> > 
> > +	blk_queue_max_sectors(drive->queue, 32);
> > +
> I am currently testing the effect on throughput and will have some
> better numbers soon.

OK, here are my bonnie test results.  They are not what I expected -
read and write performance is significantly better with 16KB max request
size, while seeking is much better with 1024KB.

jackd was running in the background in both cases.  With 1024KB, there
were massive XRUNS, and worse, occasionally the soundcard interrupt was
completely lost for tens of milliseconds.  This is what I would expect 
if huge SG lists are being built in hardirq context.  With 16KB, jackd
ran perfectly, the highest latency I was was about 100 usecs.

Kernel is 2.6.8-rc2 + voluntary-preempt-I4.  CPU is 600Mhz, 512MB RAM.

16KB:

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
mindpipe         1G  3002  88 31164  60 13485  29  3035  84 32838  36  56.7   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16    94  40 +++++ +++ 15510  66   204  86 +++++ +++   906  83 
mindpipe,1G,3002,88,31164,60,13485,29,3035,84,32838,36,56.7,0,16,94,40,+++++,+++,15510,66,204,86,+++++,+++,906,83

1024KB:

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
mindpipe         1G  2988  86 17647  29 10979  17  3114  90 28556  25 156.7   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16   210  88 +++++ +++ 13295  62   215  89 +++++ +++   920  84 
mindpipe,1G,2988,86,17647,29,10979,17,3114,90,28556,25,156.7,1,16,210,88,+++++,+++,13295,62,215,89,+++++,+++,920,84

HD info:
/dev/hdc:

 Model=Maxtor 6Y160P0, FwRev=YAR41BW0, SerialNo=Y44K8TZE
 Config={ Fixed }
 RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=57
 BuffType=DualPortCache, BuffSize=7936kB, MaxMultSect=16, MultSect=16
 CurCHS=4047/16/255, CurSects=16511760, LBA=yes, LBAsects=268435455
 IORDY=on/off, tPIO={min:120,w/IORDY:120}, tDMA={min:120,rec:120}
 PIO modes:  pio0 pio1 pio2 pio3 pio4 
 DMA modes:  mdma0 mdma1 mdma2 
 UDMA modes: udma0 udma1 udma2 udma3 udma4 udma5 *udma6 
 AdvancedPM=yes: disabled (255) WriteCache=enabled
 Drive conforms to: (null): 

 * signifies the current active mode

Lee

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Lee Revell]

On Sat, 2004-07-24 at 01:46, Lee Revell wrote:
> On Sat, 2004-07-24 at 00:07, Lee Revell wrote:
> > On Tue, 2004-07-20 at 08:19, Jens Axboe wrote:
> > > On Tue, Jul 20 2004, Ingo Molnar wrote:
> > > > > How much I/O do you allow to be in flight at once?  It seems like by
> > > > > decreasing the maximum size of I/O that you handle in one interrupt
> > > > > you could improve this quite a bit.  Disk throughput is good enough,
> > > > > anyone in the real world who would feel a 10% hit would just throw
> > > > > hardware at the problem.
> > > > 
> > > it's not tweakable right now, but if you wish to experiment you just
> > > need to add a line to ide-disk.c:idedisk_setup() - pseudo patch:
> > > 
> > > +	blk_queue_max_sectors(drive->queue, 32);
> > > +
> > I am currently testing the effect on throughput and will have some
> > better numbers soon.
> 
> OK, here are my bonnie test results.  They are not what I expected -
> read and write performance is significantly better with 16KB max request
> size, while seeking is much better with 1024KB.
> 

I repeated the test with 'bonnie -f' and without jackd running, and with
32KB vs. 1024KB.  The results are not as drastic, which suggests that
all the overhead from the XRUN tracing was a factor.

These results show that 32KB is better than 1024KB in some areas, and
not singificantly worse in any of these metrics.

32KB:

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
mindpipe         1G           38499  68 16383  28           35913  34 154.5   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16   257  99 +++++ +++ 24008  98   256  99 +++++ +++  1179  98
mindpipe,1G,,,38499,68,16383,28,,,35913,34,154.5,1,16,257,99,+++++,+++,24008,98,256,99,+++++,+++,1179,98

1024KB:

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
mindpipe         1G           38285  60 13961  20           33694  28 155.2   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16   259  99 +++++ +++ 24250  98   256  99 +++++ +++  1184  98
mindpipe,1G,,,38285,60,13961,20,,,33694,28,155.2,1,16,259,99,+++++,+++,24250,98,256,99,+++++,+++,1184,98

Lee

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Ingo Molnar]

* Lee Revell <rlrevell@joe-job.com> wrote:

> jackd was running in the background in both cases.  With 1024KB, there
> were massive XRUNS, and worse, occasionally the soundcard interrupt
> was completely lost for tens of milliseconds.  This is what I would
> expect if huge SG lists are being built in hardirq context.  With
> 16KB, jackd ran perfectly, the highest latency I was was about 100
> usecs.
> 
> Kernel is 2.6.8-rc2 + voluntary-preempt-I4.  CPU is 600Mhz, 512MB RAM.

ok, i'll put in a tunable for the sg size.

Btw., it's not really the building of the SG list that is expensive,
it's the completion activity that is expensive since e.g. in the case of
ext3 IO traffic it goes over _every single_ sg entry with the following
fat codepath:

__end_that_request_first()
  bio_endio()
    end_bio_bh_io_sync()
      journal_end_buffer_io_sync()
         unlock_buffer()
           wake_up_buffer()
    bio_put()
      bio_destructor()
        mempool_free()
          mempool_free_slab()
            kmem_cache_free()
        mempool_free()
          mempool_free_slab()
            kmem_cache_free()

the buffer_head, the bio and bio->bi_io_vec all lie on different
cachelines and are very likely to be not cached after long IO latencies. 
So we eat at least 3 big cachemisses, times 256.

Jens, one solution would be to make BIO completion a softirq - like SCSI
does. That makes the latencies much easier to control.

Another thing would be to create a compound structure for bio and
[typical sizes of] bio->bi_io_vec and free them as one entity, this
would get rid of one of the cachemisses. (there cannot be a 3-way
compound structure that includes the bh too because the bh is freed
later on by ext3.)

	Ingo

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Lee Revell]

On Sat, 2004-07-24 at 02:43, Ingo Molnar wrote:
> * Lee Revell <rlrevell@joe-job.com> wrote:
> 
> > jackd was running in the background in both cases.  With 1024KB, there
> > were massive XRUNS, and worse, occasionally the soundcard interrupt
> > was completely lost for tens of milliseconds.  This is what I would
> > expect if huge SG lists are being built in hardirq context.  With
> > 16KB, jackd ran perfectly, the highest latency I was was about 100
> > usecs.
> > 
> > Kernel is 2.6.8-rc2 + voluntary-preempt-I4.  CPU is 600Mhz, 512MB RAM.
> 
> ok, i'll put in a tunable for the sg size.
> 

I tested this with every power of two from 16KB to 1024KB.  The current
default on my system is 1024KB.  This may be affected by disk controller
or other factors, someone else reported a default of 128KB with the same
drive.  Using jackd at the lowest reasonable latency setting (32
frames/period at 48000 frames/sec = 666 usecs/period ), the highest
value that does not cause problems is 256KB.  The maximum latency spike
I saw with this setting was ~220 usecs.

For anyone unfamiliar, if jackd detects that it has been delayed by more
than half a period it considers this an error condition because even
though there was not an XRUN, it probably does not have enough time left
to process a block of audio, and even if it did, it would probably not
get scheduled in time to deliver it (IOW there would be an XRUN on the
next write) so it restarts.

I therefore propose 256KB as a good default for a low latency system
once this is made tunable.  This might be a good default for any system,
I am not sure under what conditions this would be a bottleneck, but for
example this would let you handle multiple NFS clients using 8K or even
32K block sizes without compromising latency.  For a single-user,
audio-centric system like a DAW, I would go as low as possible until you
see disk throughput start to drop, 16KB works well for me.

You would want this as high as possible for, say, a CD burner, but I
don't see the point is going as high as 1024KB for a disk drive.

Lee

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Lee Revell]

On Sat, 2004-07-24 at 18:49, Lee Revell wrote:
> On Sat, 2004-07-24 at 02:43, Ingo Molnar wrote:
> > 
> > ok, i'll put in a tunable for the sg size.
> > 
> I therefore propose 256KB as a good default

Scratch this, I got a 380 usec spike with 256KB, enough to trigger jackd
to reatart.  128KB so far has only gone as high as 185 usecs.

Lee

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Mark Lord]

Note that the method used by hdparm tends to underreport
achievable throughput somewhat, because it generally only
ever has one I/O "in flight".

Cheers
-- 
Mark Lord
(hdparm keeper & the original "Linux IDE Guy")

Lee Revell wrote:
> On Tue, 2004-07-20 at 08:19, Jens Axboe wrote:
> 
>>On Tue, Jul 20 2004, Ingo Molnar wrote:
>>
>>>>How much I/O do you allow to be in flight at once?  It seems like by
>>>>decreasing the maximum size of I/O that you handle in one interrupt
>>>>you could improve this quite a bit.  Disk throughput is good enough,
>>>>anyone in the real world who would feel a 10% hit would just throw
>>>>hardware at the problem.
...
> According to hdparm, the throughput is still quite good (42MB/sec on a 
> sub-$100 IDE drive).

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Lee Revell]

On Tue, 2004-07-20 at 08:19, Jens Axboe wrote:
> On Tue, Jul 20 2004, Ingo Molnar wrote:
> > > How much I/O do you allow to be in flight at once?  It seems like by
> > > decreasing the maximum size of I/O that you handle in one interrupt
> > > you could improve this quite a bit.  Disk throughput is good enough,
> > > anyone in the real world who would feel a 10% hit would just throw
> > > hardware at the problem.
> > 
> > i'm not sure whether this particular value (max # of sg-entries per IO
> > op) is runtime tunable. Jens? Might make sense to enable elvtune-alike
> > tunability of this value.
> 
> elvtune is long dead :-)
> 

This one occured several times while bonnie was cleaning up after
itself:

ALSA /home/rlrevell/cvs/alsa/alsa-driver/alsa-kernel/core/pcm_lib.c:169:
XRUN: pcmC0D2c
 [<c01066a7>] dump_stack+0x17/0x20
 [<de935377>] snd_pcm_period_elapsed+0x2c7/0x400 [snd_pcm]
 [<de945211>] snd_emu10k1_interrupt+0xd1/0x3c0 [snd_emu10k1]
 [<c01078c3>] handle_IRQ_event+0x33/0x60
 [<c0107c05>] do_IRQ+0xa5/0x170
 [<c0106268>] common_interrupt+0x18/0x20
 [<c0161b81>] shrink_dcache_parent+0x21/0x30
 [<c0159d69>] d_unhash+0x39/0xb0
 [<c0159e4c>] vfs_rmdir+0x6c/0x1b0
 [<c015a05f>] sys_rmdir+0xcf/0xf0
 [<c0106047>] syscall_call+0x7/0xb
ALSA /home/rlrevell/cvs/alsa/alsa-driver/alsa-kernel/core/pcm_lib.c:169:
XRUN: pcmC0D0p
 [<c01066a7>] dump_stack+0x17/0x20
 [<de935377>] snd_pcm_period_elapsed+0x2c7/0x400 [snd_pcm]
 [<de945477>] snd_emu10k1_interrupt+0x337/0x3c0 [snd_emu10k1]
 [<c01078c3>] handle_IRQ_event+0x33/0x60
 [<c0107c05>] do_IRQ+0xa5/0x170
 [<c0106268>] common_interrupt+0x18/0x20
 [<c0161b81>] shrink_dcache_parent+0x21/0x30
 [<c0159d69>] d_unhash+0x39/0xb0
 [<c0159e4c>] vfs_rmdir+0x6c/0x1b0
 [<c015a05f>] sys_rmdir+0xcf/0xf0
 [<c0106047>] syscall_call+0x7/0xb

Lee

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Lee Revell]

On Mon, 2004-07-12 at 23:59, Andrew Morton wrote:
> Oh dear, these don't make much sense.
> 
> You'll have the best chance of getting decent traces with
> CONFIG_FRAME_POINTER=y and CONFIG_4KSTACKS=n, but I'm not sure that this
> will help a lot.  Are you sure that CONFIG_PREEMPT is enabled on that
> kernel?
> 

Here are some more.  These happen when I switch from X to a text console
and back:


Jul 13 04:27:50 mindpipe kernel:
Jul 13 04:27:50 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 13 04:27:50 mindpipe kernel:  [__crc_totalram_pages+1387264/5353478] snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 13 04:27:50 mindpipe kernel:  [__crc_totalram_pages+1455070/5353478] snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 13 04:27:50 mindpipe kernel:  [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 13 04:27:50 mindpipe kernel:  [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 13 04:27:50 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 13 04:27:50 mindpipe kernel:  [do_con_write+397/1888] do_con_write+0x18d/0x760
Jul 13 04:27:50 mindpipe kernel:  [con_write+27/48] con_write+0x1b/0x30
Jul 13 04:27:50 mindpipe kernel:  [opost_block+210/400] opost_block+0xd2/0x190
Jul 13 04:27:50 mindpipe kernel:  [__alloc_pages+458/864] __alloc_pages+0x1ca/0x360
Jul 13 04:27:50 mindpipe kernel:  [write_chan+396/544] write_chan+0x18c/0x220
Jul 13 04:27:50 mindpipe kernel:  [default_wake_function+0/32] default_wake_function+0x0/0x20
Jul 13 04:27:50 mindpipe kernel:  [default_wake_function+0/32] default_wake_function+0x0/0x20
Jul 13 04:27:50 mindpipe kernel:  [tty_write+434/608] tty_write+0x1b2/0x260
Jul 13 04:27:50 mindpipe kernel:  [write_chan+0/544] write_chan+0x0/0x220
Jul 13 04:27:50 mindpipe kernel:  [vfs_write+186/256] vfs_write+0xba/0x100
Jul 13 04:27:50 mindpipe kernel:  [sys_write+45/80] sys_write+0x2d/0x50
Jul 13 04:27:50 mindpipe kernel:  [syscall_call+7/11] syscall_call+0x7/0xb
Jul 13 04:27:50 mindpipe kernel:
Jul 13 04:27:50 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 13 04:27:50 mindpipe kernel:  [__crc_totalram_pages+1387264/5353478] snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 13 04:27:50 mindpipe kernel:  [group_send_sig_info+101/144] group_send_sig_info+0x65/0x90
Jul 13 04:27:50 mindpipe kernel:  [__crc_totalram_pages+1455070/5353478] snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 13 04:27:50 mindpipe kernel:  [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 13 04:27:50 mindpipe kernel:  [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 13 04:27:50 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 13 04:27:50 mindpipe kernel:  [_mmx_memcpy+141/384] _mmx_memcpy+0x8d/0x180
Jul 13 04:27:50 mindpipe kernel:  [scrup+242/320] scrup+0xf2/0x140
Jul 13 04:27:50 mindpipe kernel:  [lf+96/112] lf+0x60/0x70
Jul 13 04:27:50 mindpipe kernel:  [do_con_trol+2907/3360] do_con_trol+0xb5b/0xd20
Jul 13 04:27:50 mindpipe kernel:  [do_con_write+1128/1888] do_con_write+0x468/0x760
Jul 13 04:27:50 mindpipe kernel:  [con_put_char+51/64] con_put_char+0x33/0x40
Jul 13 04:27:50 mindpipe kernel:  [opost+162/464] opost+0xa2/0x1d0
Jul 13 04:27:50 mindpipe kernel:  [write_chan+437/544] write_chan+0x1b5/0x220
Jul 13 04:27:50 mindpipe kernel:  [default_wake_function+0/32] default_wake_function+0x0/0x20
Jul 13 04:27:50 mindpipe kernel:  [default_wake_function+0/32] default_wake_function+0x0/0x20
Jul 13 04:27:50 mindpipe kernel:  [tty_write+434/608] tty_write+0x1b2/0x260
Jul 13 04:27:50 mindpipe kernel:  [write_chan+0/544] write_chan+0x0/0x220
Jul 13 04:27:50 mindpipe kernel:  [vfs_write+186/256] vfs_write+0xba/0x100
Jul 13 04:27:50 mindpipe kernel:  [sys_write+45/80] sys_write+0x2d/0x50
Jul 13 04:27:50 mindpipe kernel:  [syscall_call+7/11] syscall_call+0x7/0xb
Jul 13 04:27:50 mindpipe kernel:
Jul 13 04:27:50 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 13 04:27:50 mindpipe kernel:  [__crc_totalram_pages+1387264/5353478] snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 13 04:27:50 mindpipe kernel:  [__crc_totalram_pages+1455070/5353478] snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 13 04:27:50 mindpipe kernel:  [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 13 04:27:50 mindpipe kernel:  [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 13 04:27:50 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 13 04:27:50 mindpipe kernel:  [__do_softirq+48/144] __do_softirq+0x30/0x90
Jul 13 04:27:50 mindpipe kernel:  [do_softirq+53/64] do_softirq+0x35/0x40
Jul 13 04:27:50 mindpipe kernel:  [do_IRQ+261/352] do_IRQ+0x105/0x160
Jul 13 04:27:50 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 13 04:27:50 mindpipe kernel:
Jul 13 04:27:50 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 13 04:27:50 mindpipe kernel:  [__crc_totalram_pages+1387264/5353478] snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 13 04:27:50 mindpipe kernel:  [__crc_totalram_pages+1455070/5353478] snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 13 04:27:50 mindpipe kernel:  [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 13 04:27:50 mindpipe kernel:  [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 13 04:27:50 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 13 04:27:50 mindpipe kernel:  [do_con_write+1035/1888] do_con_write+0x40b/0x760
Jul 13 04:27:50 mindpipe kernel:  [con_write+27/48] con_write+0x1b/0x30
Jul 13 04:27:50 mindpipe kernel:  [opost_block+210/400] opost_block+0xd2/0x190
Jul 13 04:27:50 mindpipe kernel:  [__alloc_pages+458/864] __alloc_pages+0x1ca/0x360
Jul 13 04:27:50 mindpipe kernel:  [write_chan+396/544] write_chan+0x18c/0x220
Jul 13 04:27:50 mindpipe kernel:  [default_wake_function+0/32] default_wake_function+0x0/0x20
Jul 13 04:27:50 mindpipe kernel:  [default_wake_function+0/32] default_wake_function+0x0/0x20
Jul 13 04:27:50 mindpipe kernel:  [tty_write+434/608] tty_write+0x1b2/0x260
Jul 13 04:27:50 mindpipe kernel:  [write_chan+0/544] write_chan+0x0/0x220
Jul 13 04:27:50 mindpipe kernel:  [vfs_write+186/256] vfs_write+0xba/0x100
Jul 13 04:27:50 mindpipe kernel:  [sys_write+45/80] sys_write+0x2d/0x50
Jul 13 04:27:50 mindpipe kernel:  [syscall_call+7/11] syscall_call+0x7/0xb
Jul 13 04:27:50 mindpipe kernel:
Jul 13 04:27:50 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 13 04:27:50 mindpipe kernel:  [__crc_totalram_pages+1387264/5353478] snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 13 04:27:50 mindpipe kernel:  [__crc_totalram_pages+1455070/5353478] snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 13 04:27:50 mindpipe kernel:  [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 13 04:27:50 mindpipe kernel:  [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 13 04:27:50 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 13 04:27:50 mindpipe kernel:  [do_con_write+974/1888] do_con_write+0x3ce/0x760
Jul 13 04:27:50 mindpipe kernel:  [con_put_char+51/64] con_put_char+0x33/0x40
Jul 13 04:27:50 mindpipe kernel:  [opost+327/464] opost+0x147/0x1d0
Jul 13 04:27:50 mindpipe kernel:  [write_chan+437/544] write_chan+0x1b5/0x220
Jul 13 04:27:50 mindpipe kernel:  [default_wake_function+0/32] default_wake_function+0x0/0x20
Jul 13 04:27:50 mindpipe kernel:  [default_wake_function+0/32] default_wake_function+0x0/0x20
Jul 13 04:27:50 mindpipe kernel:  [tty_write+434/608] tty_write+0x1b2/0x260
Jul 13 04:27:50 mindpipe kernel:  [write_chan+0/544] write_chan+0x0/0x220
Jul 13 04:27:50 mindpipe kernel:  [vfs_write+186/256] vfs_write+0xba/0x100
Jul 13 04:27:50 mindpipe kernel:  [sys_write+45/80] sys_write+0x2d/0x50
Jul 13 04:27:50 mindpipe kernel:  [syscall_call+7/11] syscall_call+0x7/0xb
Jul 13 04:27:50 mindpipe kernel:
Jul 13 04:27:50 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 13 04:27:50 mindpipe kernel:  [__crc_totalram_pages+1387264/5353478] snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 13 04:27:50 mindpipe kernel:  [__crc_totalram_pages+1455070/5353478] snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 13 04:27:50 mindpipe kernel:  [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 13 04:27:50 mindpipe kernel:  [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 13 04:27:50 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 13 04:27:50 mindpipe kernel:  [do_con_write+1448/1888] do_con_write+0x5a8/0x760
Jul 13 04:27:50 mindpipe kernel:  [con_write+27/48] con_write+0x1b/0x30
Jul 13 04:27:50 mindpipe kernel:  [opost_block+210/400] opost_block+0xd2/0x190
Jul 13 04:27:50 mindpipe kernel:  [__alloc_pages+458/864] __alloc_pages+0x1ca/0x360
Jul 13 04:27:50 mindpipe kernel:  [write_chan+396/544] write_chan+0x18c/0x220
Jul 13 04:27:50 mindpipe kernel:  [default_wake_function+0/32] default_wake_function+0x0/0x20
Jul 13 04:27:50 mindpipe kernel:  [default_wake_function+0/32] default_wake_function+0x0/0x20
Jul 13 04:27:50 mindpipe kernel:  [tty_write+434/608] tty_write+0x1b2/0x260
Jul 13 04:27:50 mindpipe kernel:  [write_chan+0/544] write_chan+0x0/0x220
Jul 13 04:27:50 mindpipe kernel:  [vfs_write+186/256] vfs_write+0xba/0x100
Jul 13 04:27:50 mindpipe kernel:  [sys_write+45/80] sys_write+0x2d/0x50
Jul 13 04:27:50 mindpipe kernel:  [syscall_call+7/11] syscall_call+0x7/0xb
Jul 13 04:27:50 mindpipe kernel:

Lee

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Andrew Morton]

Lee Revell <rlrevell@joe-job.com> wrote:
>
> Jul 13 04:27:50 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
>  Jul 13 04:27:50 mindpipe kernel:  [__crc_totalram_pages+1387264/5353478] snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
>  Jul 13 04:27:50 mindpipe kernel:  [group_send_sig_info+101/144] group_send_sig_info+0x65/0x90
>  Jul 13 04:27:50 mindpipe kernel:  [__crc_totalram_pages+1455070/5353478] snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
>  Jul 13 04:27:50 mindpipe kernel:  [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
>  Jul 13 04:27:50 mindpipe kernel:  [do_IRQ+155/352] do_IRQ+0x9b/0x160
>  Jul 13 04:27:50 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
>  Jul 13 04:27:50 mindpipe kernel:  [_mmx_memcpy+141/384] _mmx_memcpy+0x8d/0x180
>  Jul 13 04:27:50 mindpipe kernel:  [scrup+242/320] scrup+0xf2/0x140
>  Jul 13 04:27:50 mindpipe kernel:  [lf+96/112] lf+0x60/0x70
>  Jul 13 04:27:50 mindpipe kernel:  [do_con_trol+2907/3360] do_con_trol+0xb5b/0xd20
>  Jul 13 04:27:50 mindpipe kernel:  [do_con_write+1128/1888] do_con_write+0x468/0x760
>  Jul 13 04:27:50 mindpipe kernel:  [con_put_char+51/64] con_put_char+0x33/0x40

framebuffer scrolling inside lock_kernel().  Tricky.  Suggest you use X or
vgacon.  You can try removing the lock_kernel() calls from do_tty_write(),
but make sure you're wearing ear protection.

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Lee Revell]

On Tue, 2004-07-13 at 04:43, Andrew Morton wrote:
> Lee Revell <rlrevell@joe-job.com> wrote:
> >
> > Jul 13 04:27:50 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
> >  Jul 13 04:27:50 mindpipe kernel:  [__crc_totalram_pages+1387264/5353478] snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
> >  Jul 13 04:27:50 mindpipe kernel:  [group_send_sig_info+101/144] group_send_sig_info+0x65/0x90
> >  Jul 13 04:27:50 mindpipe kernel:  [__crc_totalram_pages+1455070/5353478] snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
> >  Jul 13 04:27:50 mindpipe kernel:  [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
> >  Jul 13 04:27:50 mindpipe kernel:  [do_IRQ+155/352] do_IRQ+0x9b/0x160
> >  Jul 13 04:27:50 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
> >  Jul 13 04:27:50 mindpipe kernel:  [_mmx_memcpy+141/384] _mmx_memcpy+0x8d/0x180
> >  Jul 13 04:27:50 mindpipe kernel:  [scrup+242/320] scrup+0xf2/0x140
> >  Jul 13 04:27:50 mindpipe kernel:  [lf+96/112] lf+0x60/0x70
> >  Jul 13 04:27:50 mindpipe kernel:  [do_con_trol+2907/3360] do_con_trol+0xb5b/0xd20
> >  Jul 13 04:27:50 mindpipe kernel:  [do_con_write+1128/1888] do_con_write+0x468/0x760
> >  Jul 13 04:27:50 mindpipe kernel:  [con_put_char+51/64] con_put_char+0x33/0x40
> 
> framebuffer scrolling inside lock_kernel().  Tricky.  Suggest you use X or
> vgacon.  You can try removing the lock_kernel() calls from do_tty_write(),
> but make sure you're wearing ear protection.
> 

OK, I figured this was not an easy one.  I can just not do that.

Here are some more.  These result from using mplayer with ALSA OSS
emulation:

Jul 13 04:31:49 mindpipe kernel:
Jul 13 04:31:49 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 13 04:31:49 mindpipe kernel:  [__crc_totalram_pages+1387264/5353478] snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 13 04:31:49 mindpipe kernel:  [__crc_totalram_pages+1455070/5353478] snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 13 04:31:49 mindpipe kernel:  [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 13 04:31:49 mindpipe kernel:  [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 13 04:31:49 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 13 04:31:49 mindpipe kernel:  [__crc_totalram_pages+1394929/5353478] snd_pcm_format_set_silence+0x4b/0x1d0 [snd_pcm]
Jul 13 04:31:49 mindpipe kernel:  [__crc_totalram_pages+2165815/5353478] snd_pcm_oss_change_params+0x5a1/0x850 [snd_pcm_oss]
Jul 13 04:31:49 mindpipe kernel:  [__crc_totalram_pages+2166620/5353478] snd_pcm_oss_get_active_substream+0x76/0x90 [snd_pcm_oss]
Jul 13 04:31:49 mindpipe kernel:  [__crc_totalram_pages+2170556/5353478] snd_pcm_oss_get_formats+0x26/0x100 [snd_pcm_oss]
Jul 13 04:31:49 mindpipe kernel:  [kfree_skbmem+23/32] kfree_skbmem+0x17/0x20
Jul 13 04:31:49 mindpipe kernel:  [__kfree_skb+101/240] __kfree_skb+0x65/0xf0
Jul 13 04:31:49 mindpipe kernel:  [packet_rcv_spkt+380/544] packet_rcv_spkt+0x17c/0x220
Jul 13 04:31:49 mindpipe kernel:  [dev_queue_xmit_nit+179/288] dev_queue_xmit_nit+0xb3/0x120
Jul 13 04:31:49 mindpipe kernel:  [avc_has_perm_noaudit+104/512] avc_has_perm_noaudit+0x68/0x200
Jul 13 04:31:49 mindpipe kernel:  [avc_has_perm+72/96] avc_has_perm+0x48/0x60
Jul 13 04:31:49 mindpipe kernel:  [inode_has_perm+73/144] inode_has_perm+0x49/0x90
Jul 13 04:31:49 mindpipe kernel:  [__crc_totalram_pages+1377866/5353478] snd_pcm_hw_constraint_minmax+0x34/0x40 [snd_pcm]
Jul 13 04:31:49 mindpipe kernel:  [__crc_totalram_pages+1359876/5353478] snd_pcm_hw_constraints_complete+0x17e/0x290 [snd_pcm]
Jul 13 04:31:49 mindpipe kernel:  [selinux_file_ioctl+213/784] selinux_file_ioctl+0xd5/0x310
Jul 13 04:31:49 mindpipe kernel:  [__crc_totalram_pages+2174928/5353478] snd_pcm_oss_open+0x1aa/0x350 [snd_pcm_oss]
Jul 13 04:31:49 mindpipe kernel:  [soundcore_open+328/768] soundcore_open+0x148/0x300
Jul 13 04:31:49 mindpipe kernel:  [chrdev_open+233/528] chrdev_open+0xe9/0x210
Jul 13 04:31:49 mindpipe kernel:  [dentry_open+260/560] dentry_open+0x104/0x230
Jul 13 04:31:49 mindpipe kernel:  [__crc_totalram_pages+2170812/5353478] snd_pcm_oss_set_format+0x26/0x60 [snd_pcm_oss]
Jul 13 04:31:49 mindpipe kernel:  [__crc_totalram_pages+2176960/5353478] snd_pcm_oss_ioctl+0x51a/0x850 [snd_pcm_oss]
Jul 13 04:31:49 mindpipe kernel:  [sys_ioctl+256/608] sys_ioctl+0x100/0x260
Jul 13 04:31:49 mindpipe kernel:  [sys_fcntl64+87/144] sys_fcntl64+0x57/0x90
Jul 13 04:31:49 mindpipe kernel:  [syscall_call+7/11] syscall_call+0x7/0xb
Jul 13 04:31:49 mindpipe kernel:
Jul 13 04:31:49 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 13 04:31:49 mindpipe kernel:  [__crc_totalram_pages+1387264/5353478] snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 13 04:31:49 mindpipe kernel:  [__crc_totalram_pages+1455070/5353478] snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 13 04:31:49 mindpipe kernel:  [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 13 04:31:49 mindpipe kernel:  [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 13 04:31:49 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 13 04:31:49 mindpipe kernel:  [__crc_totalram_pages+1373142/5353478] snd_interval_refine+0xa0/0x1c0 [snd_pcm]
Jul 13 04:31:49 mindpipe kernel:  [__crc_totalram_pages+1357711/5353478] snd_pcm_hw_rule_div+0x49/0x50 [snd_pcm]
Jul 13 04:31:49 mindpipe kernel:  [__crc_totalram_pages+1347991/5353478] snd_pcm_hw_refine+0x371/0x4a0 [snd_pcm]
Jul 13 04:31:49 mindpipe kernel:  [__crc_totalram_pages+1383137/5353478] snd_pcm_hw_param_mask+0x3b/0x50 [snd_pcm]
Jul 13 04:31:49 mindpipe kernel:  [__crc_totalram_pages+2164618/5353478] snd_pcm_oss_change_params+0xf4/0x850 [snd_pcm_oss]
Jul 13 04:31:49 mindpipe kernel:  [chrdev_open+233/528] chrdev_open+0xe9/0x210
Jul 13 04:31:49 mindpipe kernel:  [__crc_totalram_pages+2166620/5353478] snd_pcm_oss_get_active_substream+0x76/0x90 [snd_pcm_oss]
Jul 13 04:31:49 mindpipe kernel:  [__crc_totalram_pages+2170887/5353478] snd_pcm_oss_get_format+0x11/0x30 [snd_pcm_oss]
Jul 13 04:31:49 mindpipe kernel:  [__crc_totalram_pages+2176960/5353478] snd_pcm_oss_ioctl+0x51a/0x850 [snd_pcm_oss]
Jul 13 04:31:49 mindpipe kernel:  [sys_ioctl+256/608] sys_ioctl+0x100/0x260
Jul 13 04:31:49 mindpipe kernel:  [sys_fcntl64+87/144] sys_fcntl64+0x57/0x90
Jul 13 04:31:49 mindpipe kernel:  [syscall_call+7/11] syscall_call+0x7/0xb
Jul 13 04:31:49 mindpipe kernel:

Lee

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Andrew Morton]

Lee Revell <rlrevell@joe-job.com> wrote:
>
> > framebuffer scrolling inside lock_kernel().  Tricky.  Suggest you use X or
>  > vgacon.  You can try removing the lock_kernel() calls from do_tty_write(),
>  > but make sure you're wearing ear protection.
>  > 
> 
>  OK, I figured this was not an easy one.  I can just not do that.

Why not?  You can certainly try removing those [un]lock_kernel() calls.

>  Here are some more.  These result from using mplayer with ALSA OSS
>  emulation:
> 
>  Jul 13 04:31:49 mindpipe kernel:
>  Jul 13 04:31:49 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
>  Jul 13 04:31:49 mindpipe kernel:  [__crc_totalram_pages+1387264/5353478] snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
>  Jul 13 04:31:49 mindpipe kernel:  [__crc_totalram_pages+1455070/5353478] snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
>  Jul 13 04:31:49 mindpipe kernel:  [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
>  Jul 13 04:31:49 mindpipe kernel:  [do_IRQ+155/352] do_IRQ+0x9b/0x160
>  Jul 13 04:31:49 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
>  Jul 13 04:31:49 mindpipe kernel:  [__crc_totalram_pages+1394929/5353478] snd_pcm_format_set_silence+0x4b/0x1d0 [snd_pcm]
>  Jul 13 04:31:49 mindpipe kernel:  [__crc_totalram_pages+2165815/5353478] snd_pcm_oss_change_params+0x5a1/0x850 [snd_pcm_oss]
>  Jul 13 04:31:49 mindpipe kernel:  [__crc_totalram_pages+2166620/5353478] snd_pcm_oss_get_active_substream+0x76/0x90 [snd_pcm_oss]
>  Jul 13 04:31:49 mindpipe kernel:  [__crc_totalram_pages+2170556/5353478] snd_pcm_oss_get_formats+0x26/0x100 [snd_pcm_oss]

Looks like those memcpys in snd_pcm_format_set_silence() are hurting.

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Takashi Iwai]

At Tue, 13 Jul 2004 02:00:25 -0700,
Andrew Morton wrote:
> 
> Lee Revell <rlrevell@joe-job.com> wrote:
> >
> >  Here are some more.  These result from using mplayer with ALSA OSS
> >  emulation:
> > 
> >  Jul 13 04:31:49 mindpipe kernel:
> >  Jul 13 04:31:49 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
> >  Jul 13 04:31:49 mindpipe kernel:  [__crc_totalram_pages+1387264/5353478] snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
> >  Jul 13 04:31:49 mindpipe kernel:  [__crc_totalram_pages+1455070/5353478] snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
> >  Jul 13 04:31:49 mindpipe kernel:  [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
> >  Jul 13 04:31:49 mindpipe kernel:  [do_IRQ+155/352] do_IRQ+0x9b/0x160
> >  Jul 13 04:31:49 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
> >  Jul 13 04:31:49 mindpipe kernel:  [__crc_totalram_pages+1394929/5353478] snd_pcm_format_set_silence+0x4b/0x1d0 [snd_pcm]
> >  Jul 13 04:31:49 mindpipe kernel:  [__crc_totalram_pages+2165815/5353478] snd_pcm_oss_change_params+0x5a1/0x850 [snd_pcm_oss]
> >  Jul 13 04:31:49 mindpipe kernel:  [__crc_totalram_pages+2166620/5353478] snd_pcm_oss_get_active_substream+0x76/0x90 [snd_pcm_oss]
> >  Jul 13 04:31:49 mindpipe kernel:  [__crc_totalram_pages+2170556/5353478] snd_pcm_oss_get_formats+0x26/0x100 [snd_pcm_oss]
> 
> Looks like those memcpys in snd_pcm_format_set_silence() are hurting.

Hmm, but it's not in lock at least...


Takashi

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Andrew Morton]

Takashi Iwai <tiwai@suse.de> wrote:
>
> t Tue, 13 Jul 2004 02:00:25 -0700,
>  Andrew Morton wrote:
>  > 
>  > Lee Revell <rlrevell@joe-job.com> wrote:
>  > >
>  > >  Here are some more.  These result from using mplayer with ALSA OSS
>  > >  emulation:
>  > > 
>  > >  Jul 13 04:31:49 mindpipe kernel:
>  > >  Jul 13 04:31:49 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
>  > >  Jul 13 04:31:49 mindpipe kernel:  [__crc_totalram_pages+1387264/5353478] snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
>  > >  Jul 13 04:31:49 mindpipe kernel:  [__crc_totalram_pages+1455070/5353478] snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
>  > >  Jul 13 04:31:49 mindpipe kernel:  [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
>  > >  Jul 13 04:31:49 mindpipe kernel:  [do_IRQ+155/352] do_IRQ+0x9b/0x160
>  > >  Jul 13 04:31:49 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
>  > >  Jul 13 04:31:49 mindpipe kernel:  [__crc_totalram_pages+1394929/5353478] snd_pcm_format_set_silence+0x4b/0x1d0 [snd_pcm]
>  > >  Jul 13 04:31:49 mindpipe kernel:  [__crc_totalram_pages+2165815/5353478] snd_pcm_oss_change_params+0x5a1/0x850 [snd_pcm_oss]
>  > >  Jul 13 04:31:49 mindpipe kernel:  [__crc_totalram_pages+2166620/5353478] snd_pcm_oss_get_active_substream+0x76/0x90 [snd_pcm_oss]
>  > >  Jul 13 04:31:49 mindpipe kernel:  [__crc_totalram_pages+2170556/5353478] snd_pcm_oss_get_formats+0x26/0x100 [snd_pcm_oss]
>  > 
>  > Looks like those memcpys in snd_pcm_format_set_silence() are hurting.
> 
>  Hmm, but it's not in lock at least...

It is - see chrdev_open().

You can do unlock_kernel()/lock_kernel() in soundcore_open().

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Takashi Iwai]

At Tue, 13 Jul 2004 02:25:01 -0700,
Andrew Morton wrote:
> 
> Takashi Iwai <tiwai@suse.de> wrote:
> >
> > t Tue, 13 Jul 2004 02:00:25 -0700,
> >  Andrew Morton wrote:
> >  > 
> >  > Lee Revell <rlrevell@joe-job.com> wrote:
> >  > >
> >  > >  Here are some more.  These result from using mplayer with ALSA OSS
> >  > >  emulation:
> >  > > 
> >  > >  Jul 13 04:31:49 mindpipe kernel:
> >  > >  Jul 13 04:31:49 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
> >  > >  Jul 13 04:31:49 mindpipe kernel:  [__crc_totalram_pages+1387264/5353478] snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
> >  > >  Jul 13 04:31:49 mindpipe kernel:  [__crc_totalram_pages+1455070/5353478] snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
> >  > >  Jul 13 04:31:49 mindpipe kernel:  [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
> >  > >  Jul 13 04:31:49 mindpipe kernel:  [do_IRQ+155/352] do_IRQ+0x9b/0x160
> >  > >  Jul 13 04:31:49 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
> >  > >  Jul 13 04:31:49 mindpipe kernel:  [__crc_totalram_pages+1394929/5353478] snd_pcm_format_set_silence+0x4b/0x1d0 [snd_pcm]
> >  > >  Jul 13 04:31:49 mindpipe kernel:  [__crc_totalram_pages+2165815/5353478] snd_pcm_oss_change_params+0x5a1/0x850 [snd_pcm_oss]
> >  > >  Jul 13 04:31:49 mindpipe kernel:  [__crc_totalram_pages+2166620/5353478] snd_pcm_oss_get_active_substream+0x76/0x90 [snd_pcm_oss]
> >  > >  Jul 13 04:31:49 mindpipe kernel:  [__crc_totalram_pages+2170556/5353478] snd_pcm_oss_get_formats+0x26/0x100 [snd_pcm_oss]
> >  > 
> >  > Looks like those memcpys in snd_pcm_format_set_silence() are hurting.
> > 
> >  Hmm, but it's not in lock at least...
> 
> It is - see chrdev_open().

Oh yeah...

> You can do unlock_kernel()/lock_kernel() in soundcore_open().

I remember ioctl() is also in lock_kernel()?


Takashi

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Andrew Morton]

Takashi Iwai <tiwai@suse.de> wrote:
>
> > You can do unlock_kernel()/lock_kernel() in soundcore_open().
> 
>  I remember ioctl() is also in lock_kernel()?

yes, you'll need to do unlock_kernel/lock_kernel there too.  If someone
changes the rules, or otherwise calls your ioctl withoht lock_kernel() held
it will reliably go BUG with spinlock debugging enabled, so make sure you
test with CONFIG_DEBUG_SPINLOCK=y.

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Lee Revell]

On Tue, 2004-07-13 at 05:00, Andrew Morton wrote:
> Lee Revell <rlrevell@joe-job.com> wrote:
> >
> > > framebuffer scrolling inside lock_kernel().  Tricky.  Suggest you use X or
> >  > vgacon.  You can try removing the lock_kernel() calls from do_tty_write(),
> >  > but make sure you're wearing ear protection.
> >  > 
> > 
> >  OK, I figured this was not an easy one.  I can just not do that.
> 
> Why not?  You can certainly try removing those [un]lock_kernel() calls.
> 

Maybe I missed something.  What exactly do you mean by 'make sure you're
wearing ear protection'?

Lee

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Andrew Morton]

Lee Revell <rlrevell@joe-job.com> wrote:
>
> On Tue, 2004-07-13 at 05:00, Andrew Morton wrote:
> > Lee Revell <rlrevell@joe-job.com> wrote:
> > >
> > > > framebuffer scrolling inside lock_kernel().  Tricky.  Suggest you use X or
> > >  > vgacon.  You can try removing the lock_kernel() calls from do_tty_write(),
> > >  > but make sure you're wearing ear protection.
> > >  > 
> > > 
> > >  OK, I figured this was not an easy one.  I can just not do that.
> > 
> > Why not?  You can certainly try removing those [un]lock_kernel() calls.
> > 
> 
> Maybe I missed something.  What exactly do you mean by 'make sure you're
> wearing ear protection'?
> 

It might go boom.  If it does screw up, it probably won't be very seriously
bad - maybe some display glitches.  Just an experiment.

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Lee Revell]

On Tue, 2004-07-13 at 05:29, Andrew Morton wrote:
> Lee Revell <rlrevell@joe-job.com> wrote:
> >
> > On Tue, 2004-07-13 at 05:00, Andrew Morton wrote:
> > > Lee Revell <rlrevell@joe-job.com> wrote:
> > > >
> > > > > framebuffer scrolling inside lock_kernel().  Tricky.  Suggest you use X or
> > > >  > vgacon.  You can try removing the lock_kernel() calls from do_tty_write(),
> > > >  > but make sure you're wearing ear protection.
> > > >  > 
> > > > 
> > > >  OK, I figured this was not an easy one.  I can just not do that.
> > > 
> > > Why not?  You can certainly try removing those [un]lock_kernel() calls.
> > > 
> > 
> > Maybe I missed something.  What exactly do you mean by 'make sure you're
> > wearing ear protection'?
> > 
> 
> It might go boom.  If it does screw up, it probably won't be very seriously
> bad - maybe some display glitches.  Just an experiment.

Heh, I thought you literally meant it could destroy my speakers.  I have
a pretty loud amp hooked up to this thing!  I will try that.

Meanwhile, here are some more XRUS traces from the same test, run
overnight, without any significant network activity.  I believe updatedb
may have caused these.  I left out the reiserfs related ones because
that seems to be a known issue.

Jul 13 06:22:07 mindpipe kernel:
Jul 13 06:22:07 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 13 06:22:07 mindpipe kernel:  [__crc_totalram_pages+1387264/5353478] snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 13 06:22:07 mindpipe kernel:  [do_pollfd+125/144] do_pollfd+0x7d/0x90
Jul 13 06:22:07 mindpipe kernel:  [__crc_totalram_pages+1455070/5353478] snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 13 06:22:07 mindpipe kernel:  [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 13 06:22:07 mindpipe kernel:  [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 13 06:22:07 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 13 06:22:07 mindpipe kernel:
Jul 13 06:22:07 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 13 06:22:07 mindpipe kernel:  [__crc_totalram_pages+1387264/5353478] snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 13 06:22:07 mindpipe kernel:  [__crc_totalram_pages+1455070/5353478] snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 13 06:22:07 mindpipe kernel:  [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 13 06:22:07 mindpipe kernel:  [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 13 06:22:07 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 13 06:22:07 mindpipe kernel:  [__kmalloc+110/160] __kmalloc+0x6e/0xa0
Jul 13 06:22:07 mindpipe kernel:  [sys_poll+209/560] sys_poll+0xd1/0x230
Jul 13 06:22:07 mindpipe kernel:  [__pollwait+0/160] __pollwait+0x0/0xa0
Jul 13 06:22:07 mindpipe kernel:  [syscall_call+7/11] syscall_call+0x7/0xb
Jul 13 06:22:07 mindpipe kernel:
Jul 13 06:22:19 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 13 06:22:19 mindpipe kernel:  [__crc_totalram_pages+1387264/5353478] snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 13 06:22:19 mindpipe kernel:  [__crc_totalram_pages+1455070/5353478] snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 13 06:22:19 mindpipe kernel:  [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 13 06:22:19 mindpipe kernel:  [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 13 06:22:19 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 13 06:22:19 mindpipe kernel:
Jul 13 06:26:13 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 13 06:26:13 mindpipe kernel:  [__crc_totalram_pages+1387264/5353478] snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 13 06:26:13 mindpipe kernel:  [__crc_totalram_pages+2260290/5353478] find_attr+0xdc/0x250 [ntfs]
Jul 13 06:26:13 mindpipe kernel:  [__crc_totalram_pages+1455070/5353478] snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 13 06:26:13 mindpipe kernel:  [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 13 06:26:13 mindpipe kernel:  [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 13 06:26:13 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 13 06:26:13 mindpipe kernel:  [__crc_totalram_pages+2275930/5353478] ntfs_readdir+0xd24/0xfe0 [ntfs]
Jul 13 06:26:13 mindpipe kernel:  [filldir64+0/256] filldir64+0x0/0x100
Jul 13 06:26:13 mindpipe kernel:  [vfs_readdir+133/160] vfs_readdir+0x85/0xa0
Jul 13 06:26:13 mindpipe kernel:  [filldir64+0/256] filldir64+0x0/0x100
Jul 13 06:26:13 mindpipe kernel:  [sys_getdents64+102/163] sys_getdents64+0x66/0xa3
Jul 13 06:26:13 mindpipe kernel:  [filldir64+0/256] filldir64+0x0/0x100
Jul 13 06:26:13 mindpipe kernel:  [syscall_call+7/11] syscall_call+0x7/0xb
Jul 13 06:26:13 mindpipe kernel:

Lee

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Lee Revell]

On Tue, 2004-07-13 at 05:29, Andrew Morton wrote:
> Lee Revell <rlrevell@joe-job.com> wrote:
> >
> > On Tue, 2004-07-13 at 05:00, Andrew Morton wrote:
> > > Lee Revell <rlrevell@joe-job.com> wrote:
> > > >
> > > > > framebuffer scrolling inside lock_kernel().  Tricky.  Suggest you use X or
> > > >  > vgacon.  You can try removing the lock_kernel() calls from do_tty_write(),
> > > >  > but make sure you're wearing ear protection.
> > > >  > 
> > > > 
> > > >  OK, I figured this was not an easy one.  I can just not do that.
> > > 
> > > Why not?  You can certainly try removing those [un]lock_kernel() calls.
> > > 
> > 
> > Maybe I missed something.  What exactly do you mean by 'make sure you're
> > wearing ear protection'?
> > 
> 
> It might go boom.  If it does screw up, it probably won't be very seriously
> bad - maybe some display glitches.  Just an experiment.
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

Seems to work perfectly.  No visible display glitches.

I would imagine that putting a modem on a tty, then stressing the system
and watching for line errors would be a definitive test.

--- drivers/char/tty_io.c.orig  2004-07-13 16:55:28.000000000 -0400
+++ drivers/char/tty_io.c       2004-07-13 16:55:51.000000000 -0400
@@ -684,17 +684,13 @@
                return -ERESTARTSYS;
        }
        if ( test_bit(TTY_NO_WRITE_SPLIT, &tty->flags) ) {
-               lock_kernel();
                written = write(tty, file, buf, count);
-               unlock_kernel();
        } else {
                for (;;) {
                        unsigned long size = max((unsigned long)PAGE_SIZE*2, 16384UL);
                        if (size > count)
                                size = count;
-                       lock_kernel();
                        ret = write(tty, file, buf, size);
-                       unlock_kernel();
                        if (ret <= 0)
                                break;
                        written += ret;

Lee

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Takashi Iwai]

At Mon, 12 Jul 2004 17:46:39 -0700,
Andrew Morton wrote:
> 
> Lee Revell <rlrevell@joe-job.com> wrote:
> >
> > > resierfs: yes, it's a problem.  I "fixed" it multiple times in 2.4, but the
> >  > fixes ended up breaking the fs in subtle ways and I eventually gave up.
> >  > 
> > 
> >  Interesting.  There is an overwhelming consensus amongst Linux audio
> >  folks that you should use reiserfs for low latency work.
> 
> It seems to be misplaced.  A simple make-a-zillion-teeny-files test here
> exhibits a 14 millisecond holdoff.

Yes, I measured a similar on reiserfs, too.
Creation and deletion of many small files causes high latencies.

The creation latency appears with ext3, too, but vanishes when
preemption is enabled.

The deletion peaks appear on all systems.  It's in the long loop of
shrink_dcache_parent().

IIRC, the RCU tasklet stall was another cause of latency in some very
heavy cases.


But, except for creation/deletion of zillon-files, I've not seen any
big latencies during my tests with reiserfs.  Reading/writing big
files (a few GB) are totally OK on reiserfs, all under 1.5 ms.


Takashi

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Bill Davidsen]

Andrew Morton wrote:
> Lee Revell <rlrevell@joe-job.com> wrote:
> 
>>>resierfs: yes, it's a problem.  I "fixed" it multiple times in 2.4, but the
>>
>> > fixes ended up breaking the fs in subtle ways and I eventually gave up.
>> > 
>>
>> Interesting.  There is an overwhelming consensus amongst Linux audio
>> folks that you should use reiserfs for low latency work.
> 
> 
> It seems to be misplaced.  A simple make-a-zillion-teeny-files test here
> exhibits a 14 millisecond holdoff.
> 
> 
>> Should I try ext3?
> 
> 
> ext3 is certainly better than that, but still has a couple of potential
> problem spots.  ext2 is probably the best at this time.

Does anyone have any data points on XFS in this regard? I agree that 
ext2 is faster than ext3, and ext3 probably has lower latency than 
reiser3, but I have no info at all on XFS. My JFS experience is all on 
AIX, as well, so if anyone has that info it might be interesting as well.

-- 
    -bill davidsen (davidsen@tmr.com)
"The secret to procrastination is to put things off until the
  last possible moment - but no longer"  -me

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Randy.Dunlap]

On Tue, 13 Jul 2004 18:40:30 -0400 Bill Davidsen wrote:

| Andrew Morton wrote:
| > Lee Revell <rlrevell@joe-job.com> wrote:
| > 
| >>>resierfs: yes, it's a problem.  I "fixed" it multiple times in 2.4, but the
| >>
| >> > fixes ended up breaking the fs in subtle ways and I eventually gave up.
| >> > 
| >>
| >> Interesting.  There is an overwhelming consensus amongst Linux audio
| >> folks that you should use reiserfs for low latency work.
| > 
| > 
| > It seems to be misplaced.  A simple make-a-zillion-teeny-files test here
| > exhibits a 14 millisecond holdoff.
| > 
| > 
| >> Should I try ext3?
| > 
| > 
| > ext3 is certainly better than that, but still has a couple of potential
| > problem spots.  ext2 is probably the best at this time.
| 
| Does anyone have any data points on XFS in this regard? I agree that 
| ext2 is faster than ext3, and ext3 probably has lower latency than 
| reiser3, but I have no info at all on XFS. My JFS experience is all on 
| AIX, as well, so if anyone has that info it might be interesting as well.

Yes, but not recent.  I did lots of journaling-fs testing and
workloads on 2.4.19-pre7 for LinuxWorld Aug. 2004.
Presentation is here:
  http://developer.osdl.org/rddunlap/journal_fs/lwe-jgfs.pdf

Simplified summary:

XFS fared well on (large) sequential IO workloads.
And of course, none of the journaling fs-es beat ext2, but
XFS came the closest.

At the time of that presentation, JFS was very slow.  The JFS team
was working on correctness/robustness only, not performance.
I don't know the status of that today.
It's somewhat of a shame, because on paper JFS looks like a great
filesystem (IMO).

--
~Randy

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Ingo Molnar]

* Andrew Morton <akpm@osdl.org> wrote:

> >  Should I try ext3?
> 
> ext3 is certainly better than that, but still has a couple of
> potential problem spots.  ext2 is probably the best at this time.

with the voluntary-preempt patch applied ext3 is below ~500 usecs for
all things i tried on a 2GHz CPU. Without the patch i can trigger
latencies up to milliseconds (even with CONFIG_PREEMPT) by triggering a
bigger commit stream via some large file write or a cached du / causing
a stream of atime updates. (I very much suspect that all other
journalled filesystems have similar problems and they'll need
measurements and fixing just like ext3 does.)

another bigger problem area is the VM - see my patch for details. 
pagetable zapping and page reclaim are both problematic and need fixups
even under CONFIG_PREEMPT. Doing a simple 'make -j' kernel build that
hits swap triggers these easily. (after applying my patch the latencies
go below 1msec even with a 'make -j' overload.)

	Ingo

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Lee Revell]

On Mon, 2004-07-19 at 06:29, Ingo Molnar wrote:
> * Andrew Morton <akpm@osdl.org> wrote:
> 
> > >  Should I try ext3?
> > 
> > ext3 is certainly better than that, but still has a couple of
> > potential problem spots.  ext2 is probably the best at this time.
> 
> with the voluntary-preempt patch applied ext3 is below ~500 usecs for
> all things i tried on a 2GHz CPU. Without the patch i can trigger
> latencies up to milliseconds (even with CONFIG_PREEMPT) by triggering a
> bigger commit stream via some large file write or a cached du / causing
> a stream of atime updates. (I very much suspect that all other
> journalled filesystems have similar problems and they'll need
> measurements and fixing just like ext3 does.)
> 
> another bigger problem area is the VM - see my patch for details. 
> pagetable zapping and page reclaim are both problematic and need fixups
> even under CONFIG_PREEMPT. Doing a simple 'make -j' kernel build that
> hits swap triggers these easily. (after applying my patch the latencies
> go below 1msec even with a 'make -j' overload.)
> 

I discovered I can reliably produce a large XRUN by toggling Caps Lock, 
Scroll Lock, or Num Lock.  This is with 2.6.8-rc1-mm1 + voluntary
preempt (I modified the patch by hand to apply on this kernel, as
2.6.8-rc2 disables my network card).  Here is the XRUN trace.

ALSA /home/rlrevell/cvs/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
 [<c01066f7>] dump_stack+0x17/0x20
 [<de952477>] snd_pcm_period_elapsed+0x2c7/0x400 [snd_pcm]
 [<de962477>] snd_emu10k1_interrupt+0x337/0x3c0 [snd_emu10k1]
 [<c0107913>] handle_IRQ_event+0x33/0x60
 [<c0107c55>] do_IRQ+0xa5/0x170
 [<c01062b8>] common_interrupt+0x18/0x20
 [<c01d3f7f>] __delay+0xf/0x20
 [<c021881a>] atkbd_sendbyte+0x5a/0xa0
 [<c0218a35>] atkbd_command+0x1d5/0x200
 [<c0218bcb>] atkbd_event+0x16b/0x200
 [<c0215a95>] input_event+0x115/0x3d0
 [<c01ecaeb>] kbd_bh+0xbb/0x160
 [<c011a554>] tasklet_action+0x44/0x70
 [<c011a303>] __do_softirq+0x83/0x90
 [<c011a345>] do_softirq+0x35/0x40
 [<c0107cc5>] do_IRQ+0x115/0x170
 [<c01062b8>] common_interrupt+0x18/0x20
 [<c014b95e>] sys_read+0x2e/0x50
 [<c0106097>] syscall_call+0x7/0xb

Lee

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Andrew Morton]

Lee Revell <rlrevell@joe-job.com> wrote:
>
>  discovered I can reliably produce a large XRUN by toggling Caps Lock, 
> Scroll Lock, or Num Lock.  This is with 2.6.8-rc1-mm1 + voluntary
> preempt

That's odd.  I wonder if the hardware is sick.  What is the duration is the
underrun?  The info you sent didn't include that.

> (I modified the patch by hand to apply on this kernel, as
> 2.6.8-rc2 disables my network card).

eh?  That's a rather more serious problem.  Does the via-rhine.c from
2.6.8-rc1-mm1 work OK if you move it into 2.6.8-rc2?


> 
> ALSA /home/rlrevell/cvs/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
>  [<c01066f7>] dump_stack+0x17/0x20
>  [<de952477>] snd_pcm_period_elapsed+0x2c7/0x400 [snd_pcm]
>  [<de962477>] snd_emu10k1_interrupt+0x337/0x3c0 [snd_emu10k1]
>  [<c0107913>] handle_IRQ_event+0x33/0x60
>  [<c0107c55>] do_IRQ+0xa5/0x170
>  [<c01062b8>] common_interrupt+0x18/0x20
>  [<c01d3f7f>] __delay+0xf/0x20
>  [<c021881a>] atkbd_sendbyte+0x5a/0xa0
>  [<c0218a35>] atkbd_command+0x1d5/0x200
>  [<c0218bcb>] atkbd_event+0x16b/0x200
>  [<c0215a95>] input_event+0x115/0x3d0
>  [<c01ecaeb>] kbd_bh+0xbb/0x160
>  [<c011a554>] tasklet_action+0x44/0x70
>  [<c011a303>] __do_softirq+0x83/0x90
>  [<c011a345>] do_softirq+0x35/0x40
>  [<c0107cc5>] do_IRQ+0x115/0x170
>  [<c01062b8>] common_interrupt+0x18/0x20
>  [<c014b95e>] sys_read+0x2e/0x50

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Lee Revell]

On Wed, 2004-07-21 at 03:03, Andrew Morton wrote:
> Lee Revell <rlrevell@joe-job.com> wrote:
> >
> >  discovered I can reliably produce a large XRUN by toggling Caps Lock, 
> > Scroll Lock, or Num Lock.  This is with 2.6.8-rc1-mm1 + voluntary
> > preempt
> 
> That's odd.  I wonder if the hardware is sick.  What is the duration is the
> underrun?  The info you sent didn't include that.
> 

I don't think so, it's a Dell QuietKey PS/2 keyboard, plugged into a
regular PS/2 port.  The XRUN is 24-70ms, usually about 32ms.

> > (I modified the patch by hand to apply on this kernel, as
> > 2.6.8-rc2 disables my network card).
> 
> eh?  That's a rather more serious problem.  Does the via-rhine.c from
> 2.6.8-rc1-mm1 work OK if you move it into 2.6.8-rc2?
> 

I posted this to LKML, the error I get is 'Invalid MAC Address'.  I will
try the older version.

Lee
> 
> > 
> > ALSA /home/rlrevell/cvs/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
> >  [<c01066f7>] dump_stack+0x17/0x20
> >  [<de952477>] snd_pcm_period_elapsed+0x2c7/0x400 [snd_pcm]
> >  [<de962477>] snd_emu10k1_interrupt+0x337/0x3c0 [snd_emu10k1]
> >  [<c0107913>] handle_IRQ_event+0x33/0x60
> >  [<c0107c55>] do_IRQ+0xa5/0x170
> >  [<c01062b8>] common_interrupt+0x18/0x20
> >  [<c01d3f7f>] __delay+0xf/0x20
> >  [<c021881a>] atkbd_sendbyte+0x5a/0xa0
> >  [<c0218a35>] atkbd_command+0x1d5/0x200
> >  [<c0218bcb>] atkbd_event+0x16b/0x200
> >  [<c0215a95>] input_event+0x115/0x3d0
> >  [<c01ecaeb>] kbd_bh+0xbb/0x160
> >  [<c011a554>] tasklet_action+0x44/0x70
> >  [<c011a303>] __do_softirq+0x83/0x90
> >  [<c011a345>] do_softirq+0x35/0x40
> >  [<c0107cc5>] do_IRQ+0x115/0x170
> >  [<c01062b8>] common_interrupt+0x18/0x20
> >  [<c014b95e>] sys_read+0x2e/0x50
> 

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Ingo Molnar]

* Andrew Morton <akpm@osdl.org> wrote:

> >  discovered I can reliably produce a large XRUN by toggling Caps Lock, 
> > Scroll Lock, or Num Lock.  This is with 2.6.8-rc1-mm1 + voluntary
> > preempt
> 
> That's odd.  I wonder if the hardware is sick.  What is the duration is the
> underrun?  The info you sent didn't include that.

no, it's the ps2 driver that is sick. The ps2 keyboard driver is one of
the few places that busy-polls for IRQ completion from within a tasklet
context for things like led switching (yuck) - this can cause many
millisecs delays on all boards i tried.

	Ingo

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Lee Revell]

On Wed, 2004-07-21 at 01:30, Ingo Molnar wrote:
> * Andrew Morton <akpm@osdl.org> wrote:
> 
> > >  discovered I can reliably produce a large XRUN by toggling Caps Lock, 
> > > Scroll Lock, or Num Lock.  This is with 2.6.8-rc1-mm1 + voluntary
> > > preempt
> > 
> > That's odd.  I wonder if the hardware is sick.  What is the duration is the
> > underrun?  The info you sent didn't include that.
> 
> no, it's the ps2 driver that is sick. The ps2 keyboard driver is one of
> the few places that busy-polls for IRQ completion from within a tasklet
> context for things like led switching (yuck) - this can cause many
> millisecs delays on all boards i tried.
> 

I can also add that La Monte H. P. Yarroll's patch to daemonize softirqs
seems to provide major improvements in latency (does not help this
problem of course).  There has been at least one other patch posted to
LKML that that does the same thing.  Will this feature be in the kernel
anytime soon?

Lee

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Ingo Molnar]

* Lee Revell <rlrevell@joe-job.com> wrote:

> I can also add that La Monte H. P. Yarroll's patch to daemonize
> softirqs seems to provide major improvements in latency (does not help
> this problem of course).  There has been at least one other patch
> posted to LKML that that does the same thing.  Will this feature be in
> the kernel anytime soon?

we already 'daemonize' softirqs in the stock kernel if the load is high
enough. (this is what ksoftirqd does) So the only question is a tunable
to make this deferring of softirq load mandatory. Yarroll's patch is
quite complex, i dont think that is necessary. It also has at least one
conceptual problem, the NOP-ing of local_bh_disable/enable in case of
CONFIG_SOFTIRQ_THREADS is clearly wrong. Yarroll?

I've added a very simple solution to daemonize softirqs similar to
Yarroll's patch to the -H5 version of voluntary-preempt:

 http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8-rc2-H5

the 'level' of preemption is controlled via the value of
voluntary_preempt - a level of 2 (default) means that softirq-deferring
is done too. A level of 1 means only the extra preemption points are
activated, level 0 means the vanilla scheduling and softirq behavior.

below i've also attached a softirq.c patch against 2.6.8-rc2 that does
unconditional deferring. (this patch is of course not intended to be
merged upstream as-is, since normally we want to process softirqs right
after the irq context.)

	Ingo

--- linux/kernel/softirq.c.orig	
+++ linux/kernel/softirq.c	
@@ -70,7 +70,7 @@ static inline void wakeup_softirqd(void)
  */
 #define MAX_SOFTIRQ_RESTART 10
 
-asmlinkage void __do_softirq(void)
+asmlinkage void ___do_softirq(void)
 {
 	struct softirq_action *h;
 	__u32 pending;
@@ -106,6 +106,23 @@ restart:
 	__local_bh_enable();
 }
 
+asmlinkage void __do_softirq(void)
+{
+	/*
+	 * 'preempt harder'. Push all softirq processing off to ksoftirqd.
+	 */
+	if (local_softirq_pending())
+		wakeup_softirqd();
+}
+
+asmlinkage void _do_softirq(void)
+{
+	local_irq_disable();
+	___do_softirq();
+	local_irq_enable();
+}
+
+
 #ifndef __ARCH_HAS_DO_SOFTIRQ
 
 asmlinkage void do_softirq(void)
@@ -340,7 +363,7 @@ static int ksoftirqd(void * __bind_cpu)
 			preempt_disable();
 			if (cpu_is_offline((long)__bind_cpu))
 				goto wait_to_die;
-			do_softirq();
+			_do_softirq();
 			preempt_enable();
 			cond_resched();
 		}

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Ingo Molnar]

> below i've also attached a softirq.c patch against 2.6.8-rc2 that does
> unconditional deferring. (this patch is of course not intended to be
> merged upstream as-is, since normally we want to process softirqs
> right after the irq context.)

i've got a more complete patch against vanilla 2.6.8-rc2:

 http://redhat.com/~mingo/voluntary-preempt/defer-softirqs-2.6.8-rc2-A2

which introduces the following tunable:

    /proc/sys/kernel/defer_softirqs  (default: 0)

this, if enabled, causes all softirqs to be processed within ksoftirqd,
and it also breaks out of the softirq loop if preemption of ksoftirqd
has been triggered by a higher-prio task.

I've also added this additional break-out to the -H6 patch of
voluntary-preempt:

  http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8-rc2-H6

it's enabled by default.

	Ingo

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Nick Piggin]

[-- Attachment #1: Type: text/plain, Size: 1176 bytes --]

Ingo Molnar wrote:
>>below i've also attached a softirq.c patch against 2.6.8-rc2 that does
>>unconditional deferring. (this patch is of course not intended to be
>>merged upstream as-is, since normally we want to process softirqs
>>right after the irq context.)
> 
> 
> i've got a more complete patch against vanilla 2.6.8-rc2:
> 
>  http://redhat.com/~mingo/voluntary-preempt/defer-softirqs-2.6.8-rc2-A2
> 
> which introduces the following tunable:
> 
>     /proc/sys/kernel/defer_softirqs  (default: 0)
> 
> this, if enabled, causes all softirqs to be processed within ksoftirqd,
> and it also breaks out of the softirq loop if preemption of ksoftirqd
> has been triggered by a higher-prio task.
> 
> I've also added this additional break-out to the -H6 patch of
> voluntary-preempt:
> 
>   http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8-rc2-H6
> 
> it's enabled by default.
> 

What do you think about deferring softirqs just while in critical
sections?

I'm not sure how well this works, and it is CONFIG_PREEMPT only
but in theory it should prevent unbounded softirqs while under
locks without taking the performance hit of doing the context
switch.

[-- Attachment #2: lat-softirq.patch --]
[-- Type: text/x-patch, Size: 3450 bytes --]




---

 linux-2.6-npiggin/include/asm-i386/hardirq.h |    8 ++++++--
 linux-2.6-npiggin/include/linux/preempt.h    |    6 +-----
 linux-2.6-npiggin/kernel/sched.c             |   13 +++++++++++--
 linux-2.6-npiggin/kernel/softirq.c           |    4 ++--
 4 files changed, 20 insertions(+), 11 deletions(-)

diff -puN kernel/softirq.c~lat-softirq kernel/softirq.c
--- linux-2.6/kernel/softirq.c~lat-softirq	2004-07-17 00:32:52.000000000 +1000
+++ linux-2.6-npiggin/kernel/softirq.c	2004-07-17 00:32:52.000000000 +1000
@@ -68,7 +68,7 @@ static inline void wakeup_softirqd(void)
  * we want to handle softirqs as soon as possible, but they
  * should not be able to lock up the box.
  */
-#define MAX_SOFTIRQ_RESTART 10
+#define MAX_SOFTIRQ_RESTART 4
 
 asmlinkage void __do_softirq(void)
 {
@@ -322,7 +322,7 @@ void __init softirq_init(void)
 
 static int ksoftirqd(void * __bind_cpu)
 {
-	set_user_nice(current, 19);
+	set_user_nice(current, 10);
 	current->flags |= PF_NOFREEZE;
 
 	set_current_state(TASK_INTERRUPTIBLE);
diff -puN include/asm-i386/hardirq.h~lat-softirq include/asm-i386/hardirq.h
--- linux-2.6/include/asm-i386/hardirq.h~lat-softirq	2004-07-17 00:32:52.000000000 +1000
+++ linux-2.6-npiggin/include/asm-i386/hardirq.h	2004-07-17 00:32:52.000000000 +1000
@@ -80,17 +80,21 @@ typedef struct {
 # include <linux/smp_lock.h>
 # define in_atomic()	((preempt_count() & ~PREEMPT_ACTIVE) != kernel_locked())
 # define IRQ_EXIT_OFFSET (HARDIRQ_OFFSET-1)
+#define irq_exit()							\
+do {									\
+		preempt_count() -= IRQ_EXIT_OFFSET;			\
+		preempt_enable_no_resched(); /* This will catch the softirq */ \
+} while (0)
 #else
 # define in_atomic()	(preempt_count() != 0)
 # define IRQ_EXIT_OFFSET HARDIRQ_OFFSET
-#endif
 #define irq_exit()							\
 do {									\
 		preempt_count() -= IRQ_EXIT_OFFSET;			\
 		if (!in_interrupt() && softirq_pending(smp_processor_id())) \
 			do_softirq();					\
-		preempt_enable_no_resched();				\
 } while (0)
+#endif
 
 #ifndef CONFIG_SMP
 # define synchronize_irq(irq)	barrier()
diff -puN include/linux/preempt.h~lat-softirq include/linux/preempt.h
--- linux-2.6/include/linux/preempt.h~lat-softirq	2004-07-17 00:32:52.000000000 +1000
+++ linux-2.6-npiggin/include/linux/preempt.h	2004-07-17 00:32:52.000000000 +1000
@@ -16,12 +16,8 @@ do { \
 	preempt_count()++; \
 } while (0)
 
-#define dec_preempt_count() \
-do { \
-	preempt_count()--; \
-} while (0)
-
 #ifdef CONFIG_PREEMPT
+void dec_preempt_count(void);
 
 asmlinkage void preempt_schedule(void);
 
diff -puN kernel/sched.c~lat-softirq kernel/sched.c
--- linux-2.6/kernel/sched.c~lat-softirq	2004-07-17 00:32:52.000000000 +1000
+++ linux-2.6-npiggin/kernel/sched.c	2004-07-17 00:32:52.000000000 +1000
@@ -3968,8 +3968,16 @@ void __might_sleep(char *file, int line)
 EXPORT_SYMBOL(__might_sleep);
 #endif
 
+#ifdef CONFIG_PREEMPT
+void dec_preempt_count(void)
+{
+	if (unlikely(preempt_count() == 1 &&
+				softirq_pending(smp_processor_id())))
+		do_softirq();
+	preempt_count()--;
+}
 
-#if defined(CONFIG_SMP) && defined(CONFIG_PREEMPT)
+#ifdef CONFIG_SMP
 /*
  * This could be a long-held lock.  If another CPU holds it for a long time,
  * and that CPU is not asked to reschedule then *this* CPU will spin on the
@@ -4012,4 +4020,5 @@ void __sched __preempt_write_lock(rwlock
 }
 
 EXPORT_SYMBOL(__preempt_write_lock);
-#endif /* defined(CONFIG_SMP) && defined(CONFIG_PREEMPT) */
+#endif /* CONFIG_SMP */
+#endif /* CONFIG_PREEMPT */

_

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Ingo Molnar]

* Nick Piggin <nickpiggin@yahoo.com.au> wrote:

> What do you think about deferring softirqs just while in critical
> sections?
> 
> I'm not sure how well this works, and it is CONFIG_PREEMPT only but in
> theory it should prevent unbounded softirqs while under locks without
> taking the performance hit of doing the context switch.

i dont think this is sufficient. A high-prio RT task might be performing
something that is important to it but isnt in any critical section. This
includes userspace processing. We dont want to delay it with softirqs.

	Ingo

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Nick Piggin]

Ingo Molnar wrote:
> * Nick Piggin <nickpiggin@yahoo.com.au> wrote:
> 
> 
>>What do you think about deferring softirqs just while in critical
>>sections?
>>
>>I'm not sure how well this works, and it is CONFIG_PREEMPT only but in
>>theory it should prevent unbounded softirqs while under locks without
>>taking the performance hit of doing the context switch.
> 
> 
> i dont think this is sufficient. A high-prio RT task might be performing
> something that is important to it but isnt in any critical section. This
> includes userspace processing. We dont want to delay it with softirqs.
> 

Given that we're looking for something acceptable for 2.6, how about
adding
if (rt_task(current))
	kick ksoftirqd instead

Otherwise, what is the performance penalty of doing all softirq
processing from ksoftirqd?

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Ingo Molnar]

* Nick Piggin <nickpiggin@yahoo.com.au> wrote:

> Given that we're looking for something acceptable for 2.6, how about
> adding
>
> if (rt_task(current))
> 	kick ksoftirqd instead
> 
> Otherwise, what is the performance penalty of doing all softirq
> processing from ksoftirqd?

this is insufficient too. An RT task might be _waiting to run_ and
spending our time in a non-RT context (including the idle task) doing
softirq processing might delay it indefinitely.

what we could do is to add a rq->nr_running_rt and do the deferred
softirq processing unconditionally if (rq->nr_running_rt). I'd still add
a sysctl to make it unconditional for user processes too - if someone
really cares about latency and doesnt want to make all his tasks RT. 
I'll code this up for the next version of the patch.

	Ingo

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Nick Piggin]

Ingo Molnar wrote:
> * Nick Piggin <nickpiggin@yahoo.com.au> wrote:
> 
> 
>>Given that we're looking for something acceptable for 2.6, how about
>>adding
>>
>>if (rt_task(current))
>>	kick ksoftirqd instead
>>
>>Otherwise, what is the performance penalty of doing all softirq
>>processing from ksoftirqd?
> 
> 
> this is insufficient too. An RT task might be _waiting to run_ and
> spending our time in a non-RT context (including the idle task) doing
> softirq processing might delay it indefinitely.
> 
> what we could do is to add a rq->nr_running_rt and do the deferred
> softirq processing unconditionally if (rq->nr_running_rt). I'd still add
> a sysctl to make it unconditional for user processes too - if someone
> really cares about latency and doesnt want to make all his tasks RT. 
> I'll code this up for the next version of the patch.
> 

Or just if (rt_task || need_resched). Another thing that may be
worthwhile thinking about is allowing softirqs to be run directly
from interrupt if the idle thread is running, maybe with an early
exit if something becomes runnable. Although this all may be going
down the path of too much complexity.

Maybe just the simple approach of "if you care about hard latency
then let ksoftirq do everything" is best?

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Ingo Molnar]

* Nick Piggin <nickpiggin@yahoo.com.au> wrote:

> >this is insufficient too. An RT task might be _waiting to run_ and
> >spending our time in a non-RT context (including the idle task) doing
> >softirq processing might delay it indefinitely.
> >
> >what we could do is to add a rq->nr_running_rt and do the deferred
> >softirq processing unconditionally if (rq->nr_running_rt). I'd still add
> >a sysctl to make it unconditional for user processes too - if someone
> >really cares about latency and doesnt want to make all his tasks RT. 
> >I'll code this up for the next version of the patch.
> >
> 
> Or just if (rt_task || need_resched). Another thing that may be
> worthwhile thinking about is allowing softirqs to be run directly from
> interrupt if the idle thread is running, maybe with an early exit if
> something becomes runnable. Although this all may be going down the
> path of too much complexity.

this doesnt work either: once we've committed ourselves to do an
'immediate' softirq processing pass we are risking latencies. We cannot
preempt the idle task while it's processing softirqs the same way we can
do the lock-break if they are always deferred.

Also, it's not really a performance issue if the system is idle then can
switch to ksoftirqd cheaply (lazy-TLB) and once we switch away from
ksoftirqd it costs as much as switching away from the idle task.

	Ingo

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Nick Piggin]

Ingo Molnar wrote:
> * Nick Piggin <nickpiggin@yahoo.com.au> wrote:
> 
> 
>>>this is insufficient too. An RT task might be _waiting to run_ and
>>>spending our time in a non-RT context (including the idle task) doing
>>>softirq processing might delay it indefinitely.
>>>
>>>what we could do is to add a rq->nr_running_rt and do the deferred
>>>softirq processing unconditionally if (rq->nr_running_rt). I'd still add
>>>a sysctl to make it unconditional for user processes too - if someone
>>>really cares about latency and doesnt want to make all his tasks RT. 
>>>I'll code this up for the next version of the patch.
>>>
>>
>>Or just if (rt_task || need_resched). Another thing that may be
>>worthwhile thinking about is allowing softirqs to be run directly from
>>interrupt if the idle thread is running, maybe with an early exit if
>>something becomes runnable. Although this all may be going down the
>>path of too much complexity.
> 
> 
> this doesnt work either: once we've committed ourselves to do an
> 'immediate' softirq processing pass we are risking latencies. We cannot
> preempt the idle task while it's processing softirqs the same way we can
> do the lock-break if they are always deferred.
> 

It is a preempt off region no matter where it is run. I don't see how
moving it to ksoftirqd can shorten that time any further.

> Also, it's not really a performance issue if the system is idle then can
> switch to ksoftirqd cheaply (lazy-TLB) and once we switch away from
> ksoftirqd it costs as much as switching away from the idle task.

Yeah you are probably right.

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Ingo Molnar]

* Nick Piggin <nickpiggin@yahoo.com.au> wrote:

> >this doesnt work either: once we've committed ourselves to do an
> >'immediate' softirq processing pass we are risking latencies. We cannot
> >preempt the idle task while it's processing softirqs the same way we can
> >do the lock-break if they are always deferred.
> >
> 
> It is a preempt off region no matter where it is run. I don't see how
> moving it to ksoftirqd can shorten that time any further.

look at my latest patches to see how it's done. We can preempt softirq
handlers via lock-break methods. The same method doesnt work in the idle
thread. With this method i've reduced worst-case softirq latencies from
~2-4 msecs to 100-200 usecs on a 2GHz x86 box.

	Ingo

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Nick Piggin]

Ingo Molnar wrote:
> * Nick Piggin <nickpiggin@yahoo.com.au> wrote:
> 
> 
>>>this doesnt work either: once we've committed ourselves to do an
>>>'immediate' softirq processing pass we are risking latencies. We cannot
>>>preempt the idle task while it's processing softirqs the same way we can
>>>do the lock-break if they are always deferred.
>>>
>>
>>It is a preempt off region no matter where it is run. I don't see how
>>moving it to ksoftirqd can shorten that time any further.
> 
> 
> look at my latest patches to see how it's done. We can preempt softirq
> handlers via lock-break methods. The same method doesnt work in the idle

Are you referring to this patch?
http://people.redhat.com/mingo/voluntary-preempt/defer-softirqs-2.6.8-rc2-A2

Surely something similar could easily be done for irq context softirq
processing with a patch like my earlier one? And it would prevent spilling
to ksoftirq when a RT thread isn't waiting to run.

> thread. With this method i've reduced worst-case softirq latencies from
> ~2-4 msecs to 100-200 usecs on a 2GHz x86 box.
> 

Nice numbers.

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Ingo Molnar]

* Nick Piggin <nickpiggin@yahoo.com.au> wrote:

> >look at my latest patches to see how it's done. We can preempt softirq
> >handlers via lock-break methods. The same method doesnt work in the idle
> 
> Are you referring to this patch?
> http://people.redhat.com/mingo/voluntary-preempt/defer-softirqs-2.6.8-rc2-A2

no, i'm referring to this one:

  http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8-rc2-I1

(disregard the debugging induced complexity.)

> Surely something similar could easily be done for irq context softirq
> processing with a patch like my earlier one? And it would prevent
> spilling to ksoftirq when a RT thread isn't waiting to run.

the softirq-defer patch is just the first step to enable lock-break of
softirqs - the lock-break is done in the -I1 patch.

(what patch do you refer to via 'my earlier one'? Did you mean your 'if
(rt_task())' suggestion?)

	Ingo

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Nick Piggin]

Ingo Molnar wrote:
> * Nick Piggin <nickpiggin@yahoo.com.au> wrote:
> 
> 
>>>look at my latest patches to see how it's done. We can preempt softirq
>>>handlers via lock-break methods. The same method doesnt work in the idle
>>
>>Are you referring to this patch?
>>http://people.redhat.com/mingo/voluntary-preempt/defer-softirqs-2.6.8-rc2-A2
> 
> 
> no, i'm referring to this one:
> 
>   http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8-rc2-I1
> 
> (disregard the debugging induced complexity.)
> 

OK... help me out here, you are referring to this:

+/*
+ * Preempt a softirq context if necessary:
+ */
+int cond_resched_softirq(void)
+{
+	BUG_ON(!in_softirq());
+	if (softirq_need_resched()) {
+		__local_bh_enable();
+		__cond_resched();
+		local_bh_disable();
+		return 1;
+	}
+	return 0;
+}

To break out of softirq processing, right?

You wouldn't need to do this to break out of interrupt context softirqs
because you wouldn't bother returning to it. Just hand the work off to
ksoftirqd.

The main thing I am looking at is getting low latency softirqs without
always handing them off to ksoftirqd. Not to mention that my patch
disables softirqs in critical regions, which can't be bad for scalability.

> 
>>Surely something similar could easily be done for irq context softirq
>>processing with a patch like my earlier one? And it would prevent
>>spilling to ksoftirq when a RT thread isn't waiting to run.
> 
> 
> the softirq-defer patch is just the first step to enable lock-break of
> softirqs - the lock-break is done in the -I1 patch.
> 
> (what patch do you refer to via 'my earlier one'? Did you mean your 'if
> (rt_task())' suggestion?)
> 

Well yeah, obviously it would need a bit of help before it gets there.

But I'll shut up for now because I'm not the one doing all the coding :)
It seems like you're getting good results.

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Ingo Molnar]

* Nick Piggin <nickpiggin@yahoo.com.au> wrote:

> You wouldn't need to do this to break out of interrupt context
> softirqs because you wouldn't bother returning to it. Just hand the
> work off to ksoftirqd.

this is plainly not the case. Look at eg. the net_tx_action() lock-break
i did in the -I1 patch. There we first create a private queue which we
work down. With my approach we can freely reschedule _within the loop_.
With your suggestion this is not possible.

i.e. executing a softirq in a process context gives us all the
advantages of a process context: all the local state is saved and
preserved until the preemption is done. These advantages are not there
for either immediate or idle-task-only-immediate type of softirq
processing.

	Ingo

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Nick Piggin]

Ingo Molnar wrote:
> * Nick Piggin <nickpiggin@yahoo.com.au> wrote:
> 
> 
>>You wouldn't need to do this to break out of interrupt context
>>softirqs because you wouldn't bother returning to it. Just hand the
>>work off to ksoftirqd.
> 
> 
> this is plainly not the case. Look at eg. the net_tx_action() lock-break
> i did in the -I1 patch. There we first create a private queue which we
> work down. With my approach we can freely reschedule _within the loop_.
> With your suggestion this is not possible.
> 

Sorry I missed that. Yeah if you are seeing high latencies *within*
a single softirq then my thing obviously wouldn't work.

If they're as high as a couple of ms on your 2GHz machine, then they
definitely shouldn't be processed in the interrupt path, so yeah
doing them in process context is the best thing to do.

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Scott Wood]

On Wed, Jul 21, 2004 at 09:32:46PM +1000, Nick Piggin wrote:
> What do you think about deferring softirqs just while in critical
> sections?
> 
> I'm not sure how well this works, and it is CONFIG_PREEMPT only
> but in theory it should prevent unbounded softirqs while under
> locks without taking the performance hit of doing the context
> switch.

You're still running do_softirq() with preemption disabled, which is
almost as bad as doing it under a lock.

-Scott

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Ingo Molnar]

* Scott Wood <scott@timesys.com> wrote:

> You're still running do_softirq() with preemption disabled, which is
> almost as bad as doing it under a lock.

well softirqs are designed like that. I've added extra preemption code
to the latest patch to avoid repeat processing.

	Ingo

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Scott Wood]

On Wed, Jul 21, 2004 at 08:46:50PM +0200, Ingo Molnar wrote:
> * Scott Wood <scott@timesys.com> wrote:
> 
> > You're still running do_softirq() with preemption disabled, which is
> > almost as bad as doing it under a lock.
> 
> well softirqs are designed like that. 

And those who wish to continue using them like that can.  However, in
my patch they never run with preemption disabled, which can result in
substantial latency improvement (as long as nothing else is causing
similar delays).  I see nothing in the design that *requires* them to
continue running with preemption disabled.

Likewise, interrupts are "designed" to be unpreemptible, but it is
possible to run them in their own threads so as to further reduce
sources of latency (at a throughput cost, of course).  This allows
long-held spinlocks that an interrupt handler needs to acquire to be
replaced with mutexes that don't inhibit preemption.

Of course, a better fix is to keep the interrupt handlers and
critical sections short, but threading them can be very effective for
producing low latencies in the short term (we were able to achieve
worst measured case latencies of well under 100us on ordinary PC
hardware under 2.4.x using this approach).

> I've added extra preemption code to the latest patch to avoid
> repeat processing.

Do you also add preemption checks in all of the various loops that
can be run from within a single softirq instance?

-Scott

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Ingo Molnar]

* Scott Wood <scott@timesys.com> wrote:

> > > You're still running do_softirq() with preemption disabled, which is
> > > almost as bad as doing it under a lock.
> > 
> > well softirqs are designed like that. 
> 
> And those who wish to continue using them like that can.  However, in
> my patch they never run with preemption disabled, which can result in
> substantial latency improvement (as long as nothing else is causing
> similar delays).  I see nothing in the design that *requires* them to
> continue running with preemption disabled.

did you get my other mail in which i explained how e.g. the networking
code _relies_ on the softirq semantics?

of course you can improve latencies by making something preemptible that
wasnt preemptible before, but if you do that you should be absolutely
sure that it can be done without breaking that code. And in this case it
cannot be done. The comments in your patch also suggest that you were
unsure about this issue:

 +/* As far as I can tell, local_bh_disable() didn't stop ksoftirqd
 +   from running before.  Since all softirqs now run from one of
 +   the ksoftirqds, this shouldn't be necessary. */
 +
 +static inline void local_bh_disable(void)
 +{
 +}

local_bh_disable() of course stops ksoftirqd from running on that CPU.

> Do you also add preemption checks in all of the various loops that can
> be run from within a single softirq instance?

no, that's the next step, if those loops turn out to be problematic. 
E.g. network device backlogs default to a value of 100 or so which
shouldnt generate too bad latencies. If it does it's easy to break out
of those loops, they are already breakout-aware.

	Ingo

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Scott Wood]

On Wed, Jul 21, 2004 at 11:08:32PM +0200, Ingo Molnar wrote:
> did you get my other mail in which i explained how e.g. the networking
> code _relies_ on the softirq semantics?

Yes, but not until after I sent that e-mail.

> > Do you also add preemption checks in all of the various loops that can
> > be run from within a single softirq instance?
> 
> no, that's the next step, if those loops turn out to be problematic. 
> E.g. network device backlogs default to a value of 100 or so which
> shouldnt generate too bad latencies. If it does it's easy to break out
> of those loops, they are already breakout-aware.

I suppose it depends on what sort of latencies you're trying to
achieve, and how fast the hardware you're running on is.  In our 2.4
kernel we typically achieved worst case latencies of 50us or so on PC
hardware, and a few hundred us on slower embedded hardware.  Packet
processing would have to be awfully fast to cram 100 packets in that
time...

-Scott

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Ingo Molnar]

* Scott Wood <scott@timesys.com> wrote:

> Likewise, interrupts are "designed" to be unpreemptible, but it is
> possible to run them in their own threads so as to further reduce
> sources of latency (at a throughput cost, of course).  This allows
> long-held spinlocks that an interrupt handler needs to acquire to be
> replaced with mutexes that don't inhibit preemption.
> 
> Of course, a better fix is to keep the interrupt handlers and critical
> sections short, but threading them can be very effective for producing
> low latencies in the short term (we were able to achieve worst
> measured case latencies of well under 100us on ordinary PC hardware
> under 2.4.x using this approach).

do you have a 2.6 patch for hardirq redirection too? I always thought
this to be the best approach to achive hard-RT class latency guarantees
under Linux (but never coded it up). The problem with RTLinux is that it
introduces a separate OS (with separate APIs). It is (much) further
ahead of Linux in terms of latencies, algorithms and guarantees but is
still a separate OS. I believe there is a natural synergy between low
latencies needed for good desktop and multiuser performance and
soft-RT/hard-RT needs, which we should use - RTLinux doesnt generate
this synergy.

if both hardirqs and softirqs are redirectable to process contexts then
the only unpredictable latency would be the very short IRQ entry stub of
a new hardirq costing ~5 usecs - which latency is limited in effect
unless the CPU is hopelessly bombarded with interrupts.

to solve the spinlock problem of hardirqs i'd propose a dual type
spinlock that is a spinlock if hardirqs are immediate (synchronous) and
it would be a mutex if hardirqs are redirected (asynchronous). Then some
simple driver could be converted to this RT-aware spinlock and we'd see
how well it works. Have you done experiments in this direction? I
believe this all could be merged upstream, given sufficient cleanliness.

	Ingo

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Scott Wood]

On Wed, Jul 21, 2004 at 11:45:34PM +0200, Ingo Molnar wrote:
> do you have a 2.6 patch for hardirq redirection too? 

Yes, and I'll try to get it cleaned up and posted soon.  It depends
on threaded softirqs, though (so I'll have to do something with the
local_bh_disable; probably put it back to the way it was, at least
for now).

> I always thought this to be the best approach to achive hard-RT
> class latency guarantees under Linux (but never coded it up). The
> problem with RTLinux is that it introduces a separate OS (with
> separate APIs). 

If you really want hard-RT-type guarantees, though, you'd need to go
with a scheme whereby interrupts/preemption are disabled only where
the code has specifically been "approved" as not capable of causing
excessive latency (which IRQ threads are a step towards, and which is
what we did in 2.4), as opposed to the current scheme of benchmarking
latencies and fixing those places which actually show up as being
problematic.  There's always the possibility that a different usage
case will show up later, causing the latency in some piece of code to
go way up.

> if both hardirqs and softirqs are redirectable to process contexts then
> the only unpredictable latency would be the very short IRQ entry stub of
> a new hardirq costing ~5 usecs - which latency is limited in effect
> unless the CPU is hopelessly bombarded with interrupts.

Those aren't the only sources; you also have preempt_disable() and
such (as well as hardware weirdness, though there's not much we can
do about that).

> to solve the spinlock problem of hardirqs i'd propose a dual type
> spinlock that is a spinlock if hardirqs are immediate (synchronous) and
> it would be a mutex if hardirqs are redirected (asynchronous). Then some
> simple driver could be converted to this RT-aware spinlock and we'd see
> how well it works. Have you done experiments in this direction? 

This sort of substitution is what we did in 2.4, though we made this
type the default and gave the real spinlocks a new name to be used in
those few places where it was really needed.  Of course, this
resulted in a lot of places using a mutex where a spinlock would have
been fine.

-Scott

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Ingo Molnar]

* Scott Wood <scott@timesys.com> wrote:

> > if both hardirqs and softirqs are redirectable to process contexts then
> > the only unpredictable latency would be the very short IRQ entry stub of
> > a new hardirq costing ~5 usecs - which latency is limited in effect
> > unless the CPU is hopelessly bombarded with interrupts.
> 
> Those aren't the only sources; you also have preempt_disable() and
> such (as well as hardware weirdness, though there's not much we can do
> about that).

i did not say latency sources, i said unpredictable latency sources. 
hardirq and softirq processing introduces near arbitrary latency at any
(irqs-enabled) point in the kernel. Hence they make all kernel code
unbound in essence - even the most trivial kernel code, sys_getpid().

by (optionally) moving softirqs and hardirqs to a process context we've
removed this source of uncertainty by making all processing synchronous,
and what remains are ordinary algorithmic properties of the kernel code
- which we can predict and control. I.e. with those two fixed we now
have a _chance_ to guarantee latencies.

[the only remaining source of 'latency uncertainty' is the small
asynchronous hardirq stub that would still remain. This has an effect
that can be compared to e.g. cache effects and it cannot become unbound
unless the CPU is bombarded with a very high number of interrupts.]

> > to solve the spinlock problem of hardirqs i'd propose a dual type
> > spinlock that is a spinlock if hardirqs are immediate (synchronous) and
> > it would be a mutex if hardirqs are redirected (asynchronous). Then some
> > simple driver could be converted to this RT-aware spinlock and we'd see
> > how well it works. Have you done experiments in this direction? 
> 
> This sort of substitution is what we did in 2.4, though we made this
> type the default and gave the real spinlocks a new name to be used in
> those few places where it was really needed.  Of course, this resulted
> in a lot of places using a mutex where a spinlock would have been
> fine.

what are those few places where a spinlock was really needed?

I'm a bit uneasy about making mutexes the default not due to performance
but due to e.g. some hardware being very timing-sensitive. I dont think
we can make a blanket assumption that every code path is preemptable if
all processing is synchronous and the exclusions are listened to.

	Ingo

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Scott Wood]

On Thu, Jul 22, 2004 at 09:40:34AM +0200, Ingo Molnar wrote:
> * Scott Wood <scott@timesys.com> wrote:
> > This sort of substitution is what we did in 2.4, though we made this
> > type the default and gave the real spinlocks a new name to be used in
> > those few places where it was really needed.  Of course, this resulted
> > in a lot of places using a mutex where a spinlock would have been
> > fine.
> 
> what are those few places where a spinlock was really needed?

Places that inherently cannot sleep, such as inside the scheduler,
the unthreadable part of the hard IRQ code, inside the mutex
implementation, etc.

> I'm a bit uneasy about making mutexes the default not due to performance
> but due to e.g. some hardware being very timing-sensitive. 

In practice, this didn't turn out to be an issue; most modern
hardware seems to be pretty tolerant of that (and you already have to
deal with things like interrupts getting in the way), and drivers
which do local_irq_disable() or to ensure timing will still work.

Has this sort of problem been seen with RT-Linux and such, which
would cause similar delays?

-Scott

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Bill Huey]

On Thu, Jul 22, 2004 at 02:53:08PM -0400, Scott Wood wrote:
> On Thu, Jul 22, 2004 at 09:40:34AM +0200, Ingo Molnar wrote:
> > * Scott Wood <scott@timesys.com> wrote:
> > > This sort of substitution is what we did in 2.4, though we made this
> > > type the default and gave the real spinlocks a new name to be used in
> > > those few places where it was really needed.  Of course, this resulted
> > > in a lot of places using a mutex where a spinlock would have been
> > > fine.
> > 
> > what are those few places where a spinlock was really needed?
> 
> Places that inherently cannot sleep, such as inside the scheduler,
> the unthreadable part of the hard IRQ code, inside the mutex
> implementation, etc.

Scott and Ingo,

Drivers that you might have to poll (technically a kind of hardware level
spinlock multi-device concurrency problem... Those special spins need to be
bounded by a preempt_{dis,en}able to prevent deadlocking) for completion since
it can't do an async notify of some sort, the low level timer handling
infrastructure, all places in the scheduler, all child functions/places that
can sleep/block within a non-preemptable critical section must be demoted back
to spinlocks and more stuff. The problem isn't trivial at all. The latter one
on the list is something that requires progressive audit as minor kernel
releases come out.

This is major project. kgdb needs to be ported to it, etc... and other
things that slip my mind right now.

There are also other problem with moving to a largely sleeping mutex
style kernel, dead lock detection becomes sorely needed. Current spinlock
detection methods are probably going to be useless in a system like this.
BSD/OS-FreeBSD have some of these facilites. The TimeSys mutexes have a
read/write lock, akind to BSD/OS shared/exclusive locks, that has a depth
first search cycle detector in them. My background is BSD/OS-FreeBSD.

The problem with FreeBSD is that their project is pretty much crap
in that they have these ego-manics focused on cute little SMP mechanisms
instead of actually doing the hard work of pushing locks down into lower
level subsystems. That's why DragonFly-BSD split from them in addition
to other multipule reasons. They also have tons more class and are nicer,
smarter folks in general. :)

The good thing about this, Linux, is that many of the RTOS issues we're
talking about now have a direct 1:1 relationship to SMP locking issues.
The concurrency issues, in a mathematical manner, are the same. It's not
just the trivial stuff with holding a non-preemptive lock such as BLK, but
in the context of priority inheritance and contention where, I believe,
the simple case use of priority inheritance should be considered a kind
contention overload condition of the overall system.

The general purpose OS folks like FreeBSD and old versions Solaris have a
sort of confused and backward notion of this problem by using priority
inheritance as a substitute for fixing the contention problems in the
first place. They abuse the scheduler to try and a get system that can
respond regularly to priority centric schedulers, but what happens instead
is they spread scheduling irregularities that effect the entire kernel
in unknown, unpredictable and odd ways. RTOS folks are much more sensitive
to the over and inappropriate use of things like this.

Linux is prime for some kind of RTOS conversion in that locks have been
pushed down sufficiently that late/runtime detection logic with simple
priority inheritance would rarely be triggered. That's basically a kind
of lock contention maladaptation. There's a possibility of making Linux
a top-notch RTOS if the right folks where working on that stuff.

Rant over... :)

BTW, this basically turning Linux into a kind of SGI IRIX style system.

> > I'm a bit uneasy about making mutexes the default not due to performance
> > but due to e.g. some hardware being very timing-sensitive. 
> 
> In practice, this didn't turn out to be an issue; most modern
> hardware seems to be pretty tolerant of that (and you already have to
> deal with things like interrupts getting in the way), and drivers
> which do local_irq_disable() or to ensure timing will still work.
 
> Has this sort of problem been seen with RT-Linux and such, which
> would cause similar delays?

bill

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Ingo Molnar]

* Bill Huey <bhuey@lnxw.com> wrote:

> There are also other problem with moving to a largely sleeping mutex
> style kernel, dead lock detection becomes sorely needed. Current
> spinlock detection methods are probably going to be useless in a
> system like this. [...]

i have another worry with the 'everything is a mutex' concept. Currently
we still do have a number of 'central' locks such as dcache_lock, or the
SLAB locks. So even if all (but the scheduling) spinlocks are converted
to sleeping mutexes what do you gain? A high-prio RT task will get to
execute userspace instructions almost immediately, but any kernel
functionality use of this RT thread might still be blocked by a priority
inversion problem. So the same type of latency problems that we are
detecting and solving currently will occur on a mutex-based system as
well - if the RT application wants to use kernel functionality.

so why dont we keep the spinlocks [on UP, nonpreemptible sections] and
just let the kernel finish its work and get to a fully lock-quiescent
state ASAP where we can reschedule? It's not like that a high-prio RT
task can avoid this situation with any guarantee, as long as these
central locks remain. (in fact it would have to do this processing with
higher overhead, because the lowprio thread that got preemption needs to
be boosted, scheduled, unscheduled and the high-prio task needs to
schedule again.)

i'd agree with turning most of the finegrained per-task (non-irq-safe)
spinlocks into mutexes (or spin-mutexes). But the central locks that an
RT task would likely hit need to remain spinlocks i believe.

plus there are central mutexes too that are in 'hiding' currently but
could cause latencies just as much.

	Ingo

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Scott Wood]

On Wed, Jul 28, 2004 at 08:45:47AM +0200, Ingo Molnar wrote:
> i have another worry with the 'everything is a mutex' concept. Currently
> we still do have a number of 'central' locks such as dcache_lock, or the
> SLAB locks. So even if all (but the scheduling) spinlocks are converted
> to sleeping mutexes what do you gain? A high-prio RT task will get to
> execute userspace instructions almost immediately, but any kernel
> functionality use of this RT thread might still be blocked by a priority
> inversion problem. So the same type of latency problems that we are
> detecting and solving currently will occur on a mutex-based system as
> well - if the RT application wants to use kernel functionality.

The difference is that latency issues are isolated to code which uses
the locks with bad critical sections.  If some random driver holds an
internal lock for a few milliseconds, it won't slow down anything
that doesn't interact with the driver (either directly or by blocking
on something else that interacts with the driver).  Turning the
spinlocks into mutexes doesn't eliminate the need for lock-breaking;
it just lets the lock-breaking effort be concentrated on the core
mutexes, so you don't need to fix *all* of the bad critical sections
to see a reduction in worst case latency of common code.

Those critical sections where lock-breaking has been done can be
converted back into spinlocks.  Essentially, mutexes would be used
for "untrusted" locks, as opposed to using spinlocks just where
they're absolutely necessary.  Over time, the set of trusted locks
would presumably go up, though we'd have to be careful to make sure
people know that they need to be especially careful of latency issues
when they touch code that uses such locks.

One of the main benefits is that it significantly increases the RT
guarantees for those users for whom the RT portion of their app can
be verified as only using a limited, testable/auditable subset of
kernel paths.  Otherwise, you run the risk of some special code path
(such as out-of-memory, perhaps) that didn't get hit in latency
testing showing up and causing problems for a RT task that doesn't
need memory, and doesn't go anywhere near the lock in question.  How
much this matters depends on how hard the RT one needs is.

Another is that the observed latencies of many apps is likely to go
down faster than when squashing each individual latency in the system
as it shows up.  Said squashing can (and should) still occur, but at
least some apps will not have to wait.

-Scott

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Ingo Molnar]

* Scott Wood <scott@timesys.com> wrote:

> Those critical sections where lock-breaking has been done can be
> converted back into spinlocks.  Essentially, mutexes would be used for
> "untrusted" locks, as opposed to using spinlocks just where they're
> absolutely necessary.  Over time, the set of trusted locks would
> presumably go up, though we'd have to be careful to make sure people
> know that they need to be especially careful of latency issues when
> they touch code that uses such locks.
> 
> One of the main benefits is that it significantly increases the RT
> guarantees for those users for whom the RT portion of their app can be
> verified as only using a limited, testable/auditable subset of kernel
> paths. [...]

ok, i see - this makes 100% sense. I'm wondering how intrusive such an
all-preemptive patchset is? There are some problems with per-CPU data
structures on SMP. Right now holding a spinlock means one can use
smp_processor_id() and rely on it staying constant in the critical
section. With a mutex in the same place all such assumptions would
break. Is there some automatic way to deal with these issues (or to at
least detect them reliably?).

	Ingo

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Bill Huey]

On Thu, Jul 29, 2004 at 08:21:10PM +0200, Ingo Molnar wrote:
> * Scott Wood <scott@timesys.com> wrote:
> ok, i see - this makes 100% sense. I'm wondering how intrusive such an
> all-preemptive patchset is? There are some problems with per-CPU data
> structures on SMP. Right now holding a spinlock means one can use
> smp_processor_id() and rely on it staying constant in the critical
> section. With a mutex in the same place all such assumptions would
> break. Is there some automatic way to deal with these issues (or to at
> least detect them reliably?).

Make smp_processor_id check if preempt_count() is non-zero to make sure 
that you're running within a non-preemptable critical section (scheduler
deferred). Do the same with local_irq_* critical section by checking to
see if interrupts are disabled. They are also non-preemptable (hardware
defered).

That's my suggestion.

bill

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Scott Wood]

On Thu, Jul 29, 2004 at 11:36:26AM -0700, Bill Huey wrote:
> On Thu, Jul 29, 2004 at 08:21:10PM +0200, Ingo Molnar wrote:
> > * Scott Wood <scott@timesys.com> wrote:
> > ok, i see - this makes 100% sense. I'm wondering how intrusive such an
> > all-preemptive patchset is? There are some problems with per-CPU data
> > structures on SMP. Right now holding a spinlock means one can use
> > smp_processor_id() and rely on it staying constant in the critical
> > section. With a mutex in the same place all such assumptions would
> > break. Is there some automatic way to deal with these issues (or to at
> > least detect them reliably?).
> 
> Make smp_processor_id check if preempt_count() is non-zero to make sure 
> that you're running within a non-preemptable critical section (scheduler
> deferred). Do the same with local_irq_* critical section by checking to
> see if interrupts are disabled. They are also non-preemptable (hardware
> defered).

There are legitimate reasons to use smp_processor_id() outside of a
non-preemptible region, though.  These include debugging
printks/logging, and situations where the awareness of CPU is for
optimization rather than correctness (for example, using per-cpu data
with per-cpu locks, which are only contended if preemption occurs, or
per-cpu counters incremented with atomic operations (or where counter
accuracy is not critical)).

The detection mechanism we used in 2.4 was simply to grep for
smp_processor_id and check/fix everything manually (which is somewhat
tedious, but there aren't *too* many instances in core code, and many
uses are similar to one another).

-Scott

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Bill Huey]

On Thu, Jul 29, 2004 at 03:17:52PM -0400, Scott Wood wrote:
> On Thu, Jul 29, 2004 at 11:36:26AM -0700, Bill Huey wrote:
> There are legitimate reasons to use smp_processor_id() outside of a
> non-preemptible region, though.  These include debugging
> printks/logging, and situations where the awareness of CPU is for
> optimization rather than correctness (for example, using per-cpu data
> with per-cpu locks, which are only contended if preemption occurs, or
> per-cpu counters incremented with atomic operations (or where counter
> accuracy is not critical)).
> 
> The detection mechanism we used in 2.4 was simply to grep for
> smp_processor_id and check/fix everything manually (which is somewhat
> tedious, but there aren't *too* many instances in core code, and many
> uses are similar to one another).

That's a better method. But if there's a need for a kind of runtime
detector, I guess you could do that. The use of smp_processor_id() should
be seldom enough that manually fixing all of the points really should
be the solution.

bill

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Eric St-Laurent]

On Wed, 2004-07-28 at 02:45, Ingo Molnar wrote:
> i'd agree with turning most of the finegrained per-task (non-irq-safe)
> spinlocks into mutexes (or spin-mutexes). But the central locks that an
> RT task would likely hit need to remain spinlocks i believe.
> 
> plus there are central mutexes too that are in 'hiding' currently but
> could cause latencies just as much.

Here are patches that convert spinlocks into kernel mutexes with
priority inheritance.  They look reasonably clean, and might be
interesting to try.

http://inf3-www.informatik.unibw-muenchen.de/research/linux/mutex/mutex.html


Best regards,

Eric St-Laurent

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Pavel Machek]

Hi!

> i did not say latency sources, i said unpredictable latency sources. 
> hardirq and softirq processing introduces near arbitrary latency at any
> (irqs-enabled) point in the kernel. Hence they make all kernel code
> unbound in essence - even the most trivial kernel code, sys_getpid().
...
> [the only remaining source of 'latency uncertainty' is the small
> asynchronous hardirq stub that would still remain. This has an effect
> that can be compared to e.g. cache effects and it cannot become unbound
> unless the CPU is bombarded with a very high number of interrupts.]

Well, I do not follow you I guess.

With large-enough number of hardirqs you do no progress at all.

Even if only "sane" number of irqs, if they all decide to hit within one
getpid(), this getpid is going to take quite long....
				Pavel
-- 
64 bytes from 195.113.31.123: icmp_seq=28 ttl=51 time=448769.1 ms         

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Scott Wood]

On Thu, Jul 29, 2004 at 10:26:30PM +0200, Pavel Machek wrote:
> Well, I do not follow you I guess.
> 
> With large-enough number of hardirqs you do no progress at all.
> 
> Even if only "sane" number of irqs, if they all decide to hit within one
> getpid(), this getpid is going to take quite long....
> 				Pavel

Ordinarily, yes.  However, if it's a high-priority RT task that does
the getpid(), whose priority is higher than that of the RT tasks,
you'll get at most one hardirq stub per active IRQ number; after
that, the IRQs will be masked until their threads get a chance to be
scheduled.

-Scott

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Pavel Machek]

Hi!

> > Well, I do not follow you I guess.
> > 
> > With large-enough number of hardirqs you do no progress at all.
> > 
> > Even if only "sane" number of irqs, if they all decide to hit within one
> > getpid(), this getpid is going to take quite long....
> > 				Pavel
> 
> Ordinarily, yes.  However, if it's a high-priority RT task that does
> the getpid(), whose priority is higher than that of the RT tasks,
> you'll get at most one hardirq stub per active IRQ number; after
> that, the IRQs will be masked until their threads get a chance to be
> scheduled.

But will not even num_IRQs*time_per_stub be so high that any analysis
is impractical?

...

...

Hmm, that high-priority hask only has to eat num_IRQs*time_per_stub
once, so perhaps its okay.
								Pavel

-- 
People were complaining that M$ turns users into beta-testers...
...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Ingo Molnar]

* Pavel Machek <pavel@ucw.cz> wrote:

> ...
> > [the only remaining source of 'latency uncertainty' is the small
> > asynchronous hardirq stub that would still remain. This has an effect
> > that can be compared to e.g. cache effects and it cannot become unbound
> > unless the CPU is bombarded with a very high number of interrupts.]
> 
> Well, I do not follow you I guess.
> 
> With large-enough number of hardirqs you do no progress at all.
> 
> Even if only "sane" number of irqs, if they all decide to hit within
> one getpid(), this getpid is going to take quite long....

yes, all of this assumes some _minimal_ sanity of the hardware
environment. We do detect interrupt storms and turn those IRQ sources
off, but there's no (sane) way to avoid interrupt storms from
driver-handled IRQ sources.

	Ingo

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Nick Piggin]

Scott Wood wrote:
> On Wed, Jul 21, 2004 at 09:32:46PM +1000, Nick Piggin wrote:
> 
>>What do you think about deferring softirqs just while in critical
>>sections?
>>
>>I'm not sure how well this works, and it is CONFIG_PREEMPT only
>>but in theory it should prevent unbounded softirqs while under
>>locks without taking the performance hit of doing the context
>>switch.
> 
> 
> You're still running do_softirq() with preemption disabled, which is
> almost as bad as doing it under a lock.
> 

do_softirq is only run a maximum of once with preemption disabled,
instead of an unbounded number of times.

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Scott Wood]

On Wed, Jul 21, 2004 at 10:52:46AM +0200, Ingo Molnar wrote:
> this, if enabled, causes all softirqs to be processed within ksoftirqd,
> and it also breaks out of the softirq loop if preemption of ksoftirqd
> has been triggered by a higher-prio task.

You'll still have the latency of finishing the currently executing
softirq, which often includes a loop itself (whose break condition is
based on not hogging the CPU, rather than letting higher priority
tasks in as soon as possible).

-Scott

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Ingo Molnar]

* Scott Wood <scott@timesys.com> wrote:

> On Wed, Jul 21, 2004 at 10:52:46AM +0200, Ingo Molnar wrote:
> > this, if enabled, causes all softirqs to be processed within ksoftirqd,
> > and it also breaks out of the softirq loop if preemption of ksoftirqd
> > has been triggered by a higher-prio task.
> 
> You'll still have the latency of finishing the currently executing
> softirq, which often includes a loop itself (whose break condition is
> based on not hogging the CPU, rather than letting higher priority
> tasks in as soon as possible).

well, i share your desire for lower latencies but this is what the
semantics of softirqs require.

but it's really easy to break out of softirq processing early and
restart it later. All it needs in net_rx_action() is to:

	if (softirq_defer && need_resched())
		goto softnet_break;

since the network softirq code is already latency-aware. The same
'lock-break'-alike methods can be used for other softirq code too.

	Ingo

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Scott Wood]

On Wed, Jul 21, 2004 at 10:22:18AM +0200, Ingo Molnar wrote:
> we already 'daemonize' softirqs in the stock kernel if the load is high
> enough. (this is what ksoftirqd does) So the only question is a tunable
> to make this deferring of softirq load mandatory. Yarroll's patch is
> quite complex, i dont think that is necessary.

What aspects of it do you find unnecessary?  The second thread is
needed to maintain the current high/low priority semantics (without
that, you'll either starve regular tasks with a lot of softirqs, or
starve softirqs with a busy userspace, depending on how you set the
priority of the softirq thread).

> It also has at least one
> conceptual problem, the NOP-ing of local_bh_disable/enable in case of
> CONFIG_SOFTIRQ_THREADS is clearly wrong. Yarroll?

Why is it "clearly wrong"?  As far as I can tell, the only legitimate
use of it currently is to protect against deadlock (as in
spin_lock_bh()), which is not an issue if all softirqs run from a
thread.  Ksoftirqd already ignores such disabling (unless I'm missing
something?), so code that uses it to synchronize with a softirq is
already broken.

There's also the possibility of code relying on it also being
preempt_disable(); if that's happening, it could be left alone
(though IMHO it'd be better if such code made its dependence on such
behavior explicit), with preempt_disable() being the only real
effect.

> I've added a very simple solution to daemonize softirqs similar to
> Yarroll's patch to the -H5 version of voluntary-preempt:

BTW, it was my patch; Yarroll only submitted it to the list (as he
stated at the time).

-Scott

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Ingo Molnar]

* Scott Wood <scott@timesys.com> wrote:

> > we already 'daemonize' softirqs in the stock kernel if the load is high
> > enough. (this is what ksoftirqd does) So the only question is a tunable
> > to make this deferring of softirq load mandatory. Yarroll's patch is
> > quite complex, i dont think that is necessary.
> 
> What aspects of it do you find unnecessary?  The second thread is
> needed to maintain the current high/low priority semantics (without
> that, you'll either starve regular tasks with a lot of softirqs, or
> starve softirqs with a busy userspace, depending on how you set the
> priority of the softirq thread).

what high/low semantics do you mean, other than the ordering of softirq
sources? (which is currently implemented via the __do_softirq() loop
first looking at the highest prio softirq.) So splitting up ksoftirqd
into two pieces seems like a separate issue.

> > It also has at least one
> > conceptual problem, the NOP-ing of local_bh_disable/enable in case of
> > CONFIG_SOFTIRQ_THREADS is clearly wrong. Yarroll?
> 
> Why is it "clearly wrong"?  As far as I can tell, the only legitimate
> use of it currently is to protect against deadlock (as in
> spin_lock_bh()), which is not an issue if all softirqs run from a
> thread.

local_bh_disable() excludes all softirq processing. This means that such
a section must not be preempted. E.g. the networking layer manipulates
per-CPU lists from such sections, if you remove local_bh_disable() then
from the middle of such a section we could preempt into ksoftirqd which
would break the code.

> There's also the possibility of code relying on it also being
> preempt_disable(); if that's happening, it could be left alone (though
> IMHO it'd be better if such code made its dependence on such behavior
> explicit), with preempt_disable() being the only real effect.

yes, that's how softirqs are used. The patch changes these semantics.

> > I've added a very simple solution to daemonize softirqs similar to
> > Yarroll's patch to the -H5 version of voluntary-preempt:
> 
> BTW, it was my patch; Yarroll only submitted it to the list (as he
> stated at the time).

ok - sorry about the misattribution!

	Ingo

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Scott Wood]

On Wed, Jul 21, 2004 at 08:43:08PM +0200, Ingo Molnar wrote:
> * Scott Wood <scott@timesys.com> wrote:
> > What aspects of it do you find unnecessary?  The second thread is
> > needed to maintain the current high/low priority semantics (without
> > that, you'll either starve regular tasks with a lot of softirqs, or
> > starve softirqs with a busy userspace, depending on how you set the
> > priority of the softirq thread).
> 
> what high/low semantics do you mean, other than the ordering of softirq
> sources? (which is currently implemented via the __do_softirq() loop
> first looking at the highest prio softirq.) So splitting up ksoftirqd
> into two pieces seems like a separate issue.

I meant the current split between immediate-context softirqs (which
are repesented in the patch by the high-priority ksoftirqd) and the
low-priority thread which is used to avoid starvation while allowing
softirqs to continue running if the system's otherwise more or less
idle.

> > BTW, it was my patch; Yarroll only submitted it to the list (as he
> > stated at the time).
> 
> ok - sorry about the misattribution!

It's OK; I just don't want him to be blamed for my bugs. :-)

-Scott

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Ingo Molnar]

* Scott Wood <scott@timesys.com> wrote:

> > what high/low semantics do you mean, other than the ordering of softirq
> > sources? (which is currently implemented via the __do_softirq() loop
> > first looking at the highest prio softirq.) So splitting up ksoftirqd
> > into two pieces seems like a separate issue.
> 
> I meant the current split between immediate-context softirqs (which
> are repesented in the patch by the high-priority ksoftirqd) and the
> low-priority thread which is used to avoid starvation while allowing
> softirqs to continue running if the system's otherwise more or less
> idle.

ok, i understand what you are trying to do. I dont think it makes much
sense to preserve the throttling property of the current
immediate/ksoftirqd processing. It was really an ad-hoc way to keep
softirqs from monopolizing the CPU. The simplest solution for softirq
deferral is to push it all into ksoftirqd and then let users change the
priority/policy of ksoftirqd.

it might make sense to create separate threads for each softirq level
that exists currently:

        HI_SOFTIRQ=0,
        TIMER_SOFTIRQ,
        NET_TX_SOFTIRQ,
        NET_RX_SOFTIRQ,
        SCSI_SOFTIRQ,
        TASKLET_SOFTIRQ

but this doesnt solve the problem either, because the current softirq
splitup is too opaque - there's no priority-based distinction between
softirq processing. Doing full softirq scheduling by attaching the
softirq work to the process context that generates it (or an anymous
context for things that are not connected to any particular existing
context) is way too much work and not completely possible anyway. Much
of the irq work <-> context information is lost at higher levels. We
merge IO requests and do other optimizations. To track who generated
what would be quite some work.

so since we cannot do it perfectly i'd go for the simplest approach for
now: defer to a single ksoftirqd per CPU. Then if someone comes up with
a good patch to attach all hardirq/softirq processing to a particular
context we can implement precise scheduling of hardirq/softirq work,
based on the priority/policy of the context that generated/receives the
interrupt event.

	Ingo

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Scott Wood]

On Thu, Jul 22, 2004 at 11:51:11AM +0200, Ingo Molnar wrote:
> * Scott Wood <scott@timesys.com> wrote:
> > I meant the current split between immediate-context softirqs (which
> > are repesented in the patch by the high-priority ksoftirqd) and the
> > low-priority thread which is used to avoid starvation while allowing
> > softirqs to continue running if the system's otherwise more or less
> > idle.
> 
> ok, i understand what you are trying to do. I dont think it makes much
> sense to preserve the throttling property of the current
> immediate/ksoftirqd processing. It was really an ad-hoc way to keep
> softirqs from monopolizing the CPU. The simplest solution for softirq
> deferral is to push it all into ksoftirqd and then let users change the
> priority/policy of ksoftirqd.

Ideally, yes.  However, there isn't currently a scheduling policy that
allows the softirq thread to run as a moderately high priority
realtime thread for a short period of time, and drop it to a low
priority non-realtime thread if it runs for longer than it "should".

Running it as one high priority non-realtime task would work if all
you want to run are non-realtime tasks and very high priority RT
tasks (which are intended to have higher priority than softirqs). 
This is assuming that a high-priority non-RT task will always preempt
a lower priority task except when it has depleted its timeslice; I'm
not familiar enough with the current scheduler to know whether that
is the case.

> it might make sense to create separate threads for each softirq level
> that exists currently:
> 
>         HI_SOFTIRQ=0,
>         TIMER_SOFTIRQ,
>         NET_TX_SOFTIRQ,
>         NET_RX_SOFTIRQ,
>         SCSI_SOFTIRQ,
>         TASKLET_SOFTIRQ
> 
> but this doesnt solve the problem either, because the current softirq
> splitup is too opaque - there's no priority-based distinction between
> softirq processing.

Splitting it that way would be nice from the perspective of giving
the user the ability to place more importance on certain types of
softirqs, but it's not what I was trying to do with the high/low
threads.

> so since we cannot do it perfectly i'd go for the simplest approach for
> now: defer to a single ksoftirqd per CPU.

The dual-thread model is about as simple as the immediate/thread
split that is currently there, IMHO, and while not perfect, it is an
improvement over having to pick only one policy/priority given the
current choices of policy.

-Scott

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Robert Love]

On Thu, 2004-07-22 at 14:36 -0400, Scott Wood wrote:

> This is assuming that a high-priority non-RT task will always preempt
> a lower priority task except when it has depleted its timeslice; I'm
> not familiar enough with the current scheduler to know whether that
> is the case.

Yes, this is the basic scheduling rule for non-RT: the highest priority
task, with timeslice remaining, runs.

	Robert Love

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Scott Wood]

On Wed, Jul 21, 2004 at 02:30:10PM -0400, Scott Wood wrote:
> Why is it "clearly wrong"?  As far as I can tell, the only legitimate
> use of it currently is to protect against deadlock (as in
> spin_lock_bh()), which is not an issue if all softirqs run from a
> thread.  Ksoftirqd already ignores such disabling (unless I'm missing
> something?), so code that uses it to synchronize with a softirq is
> already broken.

It appears, though, that recent kernel versions do preempt_disable()
in ksoftirqd, apparently to support CPU hotplugging[1].  When I
originally made the patch (against 2.6.0), this wasn't the case. 
Since it was done so recently, hopefully there are no cases since
then that have started depending on this behavior.

If preempt-disabled softirqs (and thus a local_bh_disable() that
works for mutual exclusion on the local CPU) become relied upon by
random pieces of kernel code, the ability to achieve very low
latencies in Linux (not counting Linux bolt-ons like RT-Linux) would
be substantially impared.  At best, we'd have to use a per-CPU mutex
to replace local_bh_disable(), causing possible performance
degradation; this would require that any other locks held at the time
are also turned into mutexes.

-Scott

[1] Supporting this along with non-preempt-disable softirqs would
require some minor changes in the way that per-CPU softirqs are done
(especially in the network code).  We did these changes in our 2.4
kernel, so as to allow priority inheritance to move a bound task
across CPUs[2].  In particular, the softirq needs to remember which CPU
it's processing data for, rather than using smp_processor_id() all
over the place, and there are a few places where a per-CPU spinlock
(only contended when the CPU binding is broken) needs to be acquired
(all of very short duration, at least in the 2.4 code).

Alternatively, some sort of "CPU reference count" could be used to
withhold permission to take a CPU offline without stopping
preemption.

[2] ...which is required for priority inheritance to work properly in
the presence of bound tasks without violating schedulability analysis
done on other CPUs.

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Ingo Molnar]

* Scott Wood <scott@timesys.com> wrote:

> It appears, though, that recent kernel versions do preempt_disable()
> in ksoftirqd, apparently to support CPU hotplugging[1].  When I
> originally made the patch (against 2.6.0), this wasn't the case. 
> Since it was done so recently, hopefully there are no cases since then
> that have started depending on this behavior.

do_softirq() always did a local_bh_disable() which stops preemption, so
softirq processing was always non-preemptible.

> If preempt-disabled softirqs (and thus a local_bh_disable() that works
> for mutual exclusion on the local CPU) become relied upon by random
> pieces of kernel code, [...]

believe me, as someone who took part in the discussions that designed
softirqs years ago and cleaned up some of it later on, i can tell you
that this property of softirqs was and is fully intentional. It's not
just some side-effect that got relied on by random code - it was used
from day one on. E.g. it enables exclusion against softirq contexts
without having to use cli/sti.

trying to make softirqs preemptible surely wont fly for 2.6 and it will
also overly complicate the softirq model. What's so terminally wrong
about adding preemption checks to the softirq paths? It should solve the
preemption problem for good. The unbound softirq paths are well-known
(mostly in the networking code) and already have preemption-alike
checks.

	Ingo

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Bill Huey]

On Wed, Jul 21, 2004 at 11:18:26PM +0200, Ingo Molnar wrote:
> trying to make softirqs preemptible surely wont fly for 2.6 and it will
> also overly complicate the softirq model. What's so terminally wrong
> about adding preemption checks to the softirq paths? It should solve the
> preemption problem for good. The unbound softirq paths are well-known
> (mostly in the networking code) and already have preemption-alike
> checks.

These folks are tring to make the entire kernel fully preemptable,
possibly, to handle arbitrary preemption at any point during the
execution. It's a noble task to make the kernel preemptable in that
way, but what I've seen is that the use of non-preemptive critical
sections commits all locks below it in the call/lock graph to also
be non-preemptive critical sections and therefore forcing the use
of traditional lock-break and other techniques to lower latency.

Adding preemption points helps with the problems, but isn't something
that can be guaranteed to have a certain latency within N numbers of
context switches and some rescheduling computations, etc...

IMO, this is something that the Linux community should think about
being friendly to or have some kind of consideration for the possibility
of this.

bill

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Bill Huey]

On Wed, Jul 21, 2004 at 03:14:23PM -0700, Bill Huey wrote:
> On Wed, Jul 21, 2004 at 11:18:26PM +0200, Ingo Molnar wrote:
> > trying to make softirqs preemptible surely wont fly for 2.6 and it will
> > also overly complicate the softirq model. What's so terminally wrong
> > about adding preemption checks to the softirq paths? It should solve the
> > preemption problem for good. The unbound softirq paths are well-known
> > (mostly in the networking code) and already have preemption-alike
> > checks.
> 
> These folks are tring to make the entire kernel fully preemptable,
> possibly, to handle arbitrary preemption at any point during the
> execution. It's a noble task to make the kernel preemptable in that
> way, but what I've seen is that the use of non-preemptive critical
> sections commits all locks below it in the call/lock graph to also
> be non-preemptive critical sections and therefore forcing the use
> of traditional lock-break and other techniques to lower latency.

One thing that would preserve the correctness of this system might be to
use a per-CPU blocking lock to directly back the uses of local_bh_*
functions explicitly. This altered system would be functionally isomorphic,
but would permit preemption in bhs.

This is something I've been thinking about for a while so it could be
off track or wrong and I'm open (pleading) for correction.

It's a radical departure from the current "deferred-everthing" model
that Linux and many other general purpose OSes are currently using.
The overall performance penalty is unknown, but the possiblity of
lowering latency could still be very significant.

bill

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Scott Wood]

On Wed, Jul 21, 2004 at 11:18:26PM +0200, Ingo Molnar wrote:
> * Scott Wood <scott@timesys.com> wrote:
> 
> > It appears, though, that recent kernel versions do preempt_disable()
> > in ksoftirqd, apparently to support CPU hotplugging[1].  When I
> > originally made the patch (against 2.6.0), this wasn't the case. 
> > Since it was done so recently, hopefully there are no cases since then
> > that have started depending on this behavior.
> 
> do_softirq() always did a local_bh_disable() which stops preemption, so
> softirq processing was always non-preemptible.

Hmm... I'm not sure how I missed that (probably by misreading the
local_irq_enable() that comes after it as a local_bh_enable()).

> believe me, as someone who took part in the discussions that designed
> softirqs years ago and cleaned up some of it later on, i can tell you
> that this property of softirqs was and is fully intentional. It's not
> just some side-effect that got relied on by random code - it was used
> from day one on. E.g. it enables exclusion against softirq contexts
> without having to use cli/sti.

It'd be nice to use locks that only exclude the specific regions in
the softirqs that are needed, but that's a lot to change at this
point...

> trying to make softirqs preemptible surely wont fly for 2.6 and it will
> also overly complicate the softirq model. What's so terminally wrong
> about adding preemption checks to the softirq paths? It should solve the
> preemption problem for good. The unbound softirq paths are well-known
> (mostly in the networking code) and already have preemption-alike
> checks.

If every such loop in every softirq is taken care of, that would work
(though only until someone adds a new softirq that forgets to check
for preemption).  I don't see any such checks in either the transmit
or receive network softirqs in vanilla 2.6.7, though (are they in a
patch, or am I overlooking them?), much less in each individual
driver.  There are checks for excessive work (where "excessive" is
not well defined in terms of actual time), but none for need_resched()
except in a few isolated places.

-Scott

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Lee Revell]

On Wed, 2004-07-21 at 03:03, Andrew Morton wrote:
> Lee Revell <rlrevell@joe-job.com> wrote:
> >
> >  discovered I can reliably produce a large XRUN by toggling Caps Lock, 
> > Scroll Lock, or Num Lock.  This is with 2.6.8-rc1-mm1 + voluntary
> > preempt
> 
> That's odd.  I wonder if the hardware is sick.  What is the duration is the
> underrun?  The info you sent didn't include that.
> 
> > (I modified the patch by hand to apply on this kernel, as
> > 2.6.8-rc2 disables my network card).
> 
> eh?  That's a rather more serious problem.  Does the via-rhine.c from
> 2.6.8-rc1-mm1 work OK if you move it into 2.6.8-rc2?

Yes, the version from 2.6.8-rc1 works, the one from 2.6.8-rc2 does not. 
It looks like 2.6.8-rc1 actually had a newer version of this file.

This one works:

         LK1.2.0-2.6 (Roger Luethi)                                                                                                                                                           
          - Massive clean-up                                                                                                                                                                   
          - Rewrite PHY, media handling (remove options, full_duplex, backoff)                                                                                                                 
          - Fix Tx engine race for good                                                                                                                                                        
                        
This one does not:
  
          LK1.1.19 (Roger Luethi)
          - Increase Tx threshold for unspecified errors

Relevant options:

CONFIG_VIA_RHINE=m
CONFIG_VIA_RHINE_MMIO=y

Lee

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Roger Luethi]

[ shorter CC: list, this is neither audio nor voluntary preempt related ]

On Wed, 21 Jul 2004 00:03:48 -0700, Andrew Morton wrote:
> > (I modified the patch by hand to apply on this kernel, as
> > 2.6.8-rc2 disables my network card).
> 
> eh?  That's a rather more serious problem.  Does the via-rhine.c from
> 2.6.8-rc1-mm1 work OK if you move it into 2.6.8-rc2?

As I mentioned previously on this list, I noticed the problem first when
I moved to 2.6.7-bk20, but when I tried to investigate, it disappeared
and never came back (thus, my summary is largely based on reports by
others). I attributed it to some intermittent hardware problem until
someone else reported it (see lkml thread: "via-rhine breaks with recent
Linus kernels : probe of 0000:00:09.0").

The problem started in mainline between 2.6.7 and 2.6.7-bk20. It does not
exist in the -mm series which contains a later version of via-rhine. IOW,
updating mainline to the latest version of via-rhine would presumably
fix the problem but it might introduce new problems and we'd never know
what the bug was and if it was actually fixed or just masked.

I hate having via-rhine broken in mainline, but I'm busy with other
stuff and I can't reproduce the problem anymore, so basically I am
waiting to hear back from Jesper Juhl who volunteered to find the patch
that makes the difference between mainline and -mm.

Roger

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Jan Depner]

Is this problem strictly with reiserfs in 2.6 or is it also a problem
with 2.4.  I actually experienced fewer xruns with reiserfs vs ext3 on
2.4 (preempt/ll) but I have no hard numbers to back that up.

Jan

On Mon, 2004-07-12 at 19:17, Lee Revell wrote:
> On Mon, 2004-07-12 at 19:31, Andrew Morton wrote:
> > Lee Revell <rlrevell@joe-job.com> wrote:
> > >
> > > On Sun, 2004-07-11 at 01:25, Andrew Morton wrote:
> > > > What we need to do is to encourage audio testers to use ALSA drivers, to
> > > > enable CONFIG_SND_DEBUG in the kernel build and to set
> > > > /proc/asound/*/*/xrun_debug and to send us the traces which result from
> > > > underruns.
> > > > 
> > > 
> > > OK, here goes.  The following traces result from running JACK overnight
> > > like so, on an otherwise idle system.  Hardware is a VIA EPIA 6000, with
> > > a 600Mhz C3 processor.  Kernel is 2.6.7 + volunatary_preempt patch. 
> > > voluntary_preempt and kernel_preemption are both on.
> > > 
> > > jackd -v --realtime -d alsa --outchannels 2 --rate 48000 --shorts
> > > --playback --period 32  --nperiods 2
> > > 
> > > These settings require less than 666 microseconds scheduler latency. 
> > > The average performance is quite good - 5-20 *microseconds*!
> > 
> > OK, thanks.  The problem areas there are the timer-based route cache
> > flushing and reiserfs.
> > 
> > We can probably fix the route caceh thing by rescheduling the timer after
> > having handled 1000 routes or whatever, although I do wonder if this is a
> > thing we really need to bother about - what else was that machine up to?
> > 
> 
> Gnutella client.  Forgot about that.  I agree, it is not reasonable to
> expect low latency with this kind of network traffic happening.  I am
> impressed it worked as well as it did.
> 
> > resierfs: yes, it's a problem.  I "fixed" it multiple times in 2.4, but the
> > fixes ended up breaking the fs in subtle ways and I eventually gave up.
> > 
> 
> Interesting.  There is an overwhelming consensus amongst Linux audio
> folks that you should use reiserfs for low latency work.  Should I try
> ext3?
> 
> Lee
> 
> 
> 
> 
> 

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Andrew Morton]

(please don't top-post)

Jan Depner <eviltwin69@cableone.net> wrote:
>
> Is this problem strictly with reiserfs in 2.6 or is it also a problem
> with 2.4.  I actually experienced fewer xruns with reiserfs vs ext3 on
> 2.4 (preempt/ll) but I have no hard numbers to back that up.

Actually, the 2.4 low-latency patch does still have some reiserfs fixes, so
it's probably better than reiserfs in 2.6.

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Florin Andrei]

On Mon, 2004-07-12 at 17:17, Lee Revell wrote:

> There is an overwhelming consensus amongst Linux audio
> folks that you should use reiserfs for low latency work.

I doubt the "overwhelming consensus" part. ReiserFS is good at
creating/deleting lots of small files and that's not a typical scenario
when doing audio work.
For audio stuff the FS does not seem to matter much. However, when
working with large files (there are audio apps that create large project
files sometimes, especially when working in 96/24 mode and that kind of
stuff) i've seen better results with XFS.

When files get really big (think: video capture) there's no doubt XFS is
best. I've had a video capture app skipping frames no matter what, on
Ext3 and ReiserFS. Switch to XFS, the result: no more frame skipping.

-- 
Florin Andrei

http://florin.myip.org/

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Paul Winkler]

On Mon, Jul 12, 2004 at 08:31:18PM -0700, Florin Andrei wrote:
> On Mon, 2004-07-12 at 17:17, Lee Revell wrote:
> 
> > There is an overwhelming consensus amongst Linux audio
> > folks that you should use reiserfs for low latency work.
> 
> I doubt the "overwhelming consensus" part. 

me too, but at some point in the 2.4 kernel cycle,  
reiserfs came out much better than ext3 in some latency tests* that 
were reported on linux-audio-dev and linux-audio-user lists.
This seems to have left many of us musicianly types with a vague 
"reiser good, ext3 bad" mindset.


* i think it was this:
http://myweb.cableone.net/eviltwin69/Arcana.html#Mark%20Knecht's%20filesystem%20tests
 
-- 

Paul Winkler
http://www.slinkp.com

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Benno Senoner]

Lee Revell wrote:

>On Mon, 2004-07-12 at 19:31, Andrew Morton wrote:
>  
>
>>
>>OK, thanks.  The problem areas there are the timer-based route cache
>>flushing and reiserfs.
>>
>>We can probably fix the route caceh thing by rescheduling the timer after
>>having handled 1000 routes or whatever, although I do wonder if this is a
>>thing we really need to bother about - what else was that machine up to?
>>
>>    
>>
>
>Gnutella client.  Forgot about that.  I agree, it is not reasonable to
>expect low latency with this kind of network traffic happening.  I am
>impressed it worked as well as it did.
>  
>

Why not reasonable ? It is very important that networking and HD I/O 
both don't interfere with low latency audio.
Think about large audio setups where you use PC hardware to act as 
dedicated samplers, software synthesizers etc.
Such clusters might be diskless and communicate with a GBit ethernet 
with a high performance file server and
in some cases lots of network I/O might occur during audio playback. So 
having latency spikes during network I/O
would be a big showstopper for many apps in certain setups.
Even ardour if run on a diskless client would need low latency while 
doing network I/O because it does lots of disk I/O
which on the diskless client translate to lots of network I/O.
Typical use could be educational or research institutions where diskless 
clients drastically lower the cost of managing large number
of boxes and allow sharing of resources. See the LTSP project.

Other low latency while high network I/O uses are video conferencing 
applications, diskless settop boxes that stream
videos over IP etc.

Therefore low level network kernel develeopers think about us poor real 
time multimedia souls :)

cheers,
Benno
http://www.linuxsampler.org

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Paul Davis]

>Other low latency while high network I/O uses are video conferencing 
>applications, diskless settop boxes that stream
>videos over IP etc.

none of these are low latency applications. 

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Lee Revell]

On Tue, 2004-07-13 at 09:26, Benno Senoner wrote:
> Lee Revell wrote:
> 
> >On Mon, 2004-07-12 at 19:31, Andrew Morton wrote:
> >  
> >
> >>
> >>OK, thanks.  The problem areas there are the timer-based route cache
> >>flushing and reiserfs.
> >>
> >>We can probably fix the route caceh thing by rescheduling the timer after
> >>having handled 1000 routes or whatever, although I do wonder if this is a
> >>thing we really need to bother about - what else was that machine up to?
> >>
> >>    
> >>
> >
> >Gnutella client.  Forgot about that.  I agree, it is not reasonable to
> >expect low latency with this kind of network traffic happening.  I am
> >impressed it worked as well as it did.
> >  
> >
> 
> Why not reasonable ? It is very important that networking and HD I/O 
> both don't interfere with low latency audio.
> Think about large audio setups where you use PC hardware to act as 
> dedicated samplers, software synthesizers etc.

Right, I did not think of these.  I just meant that you would not
typically run Gnutella on a DAW.  This should never interfere with audio
latency, as Gnutella is running at normal priority and JACK is running
SCHED_FIFO.

Besides, it looks like there is a real problem with the route cache
flushing.  I ran the same test with no significant network traffic and I
am still seeing spikes in latency.  There are all of 5 entries in the
route cache.

Kernel IP routing cache
Source          Destination     Gateway         Flags Metric Ref    Use Iface
vmail-mx.dca.ne 192.168.2.102   192.168.2.102   l     0      1      838 lo
192.168.2.102   vmail-mx.dca.ne .                     0      0        4 eth0
vmail-mx.dca.ne 192.168.2.102   192.168.2.102   l     0      1      838 lo
192.168.2.102   .               .                     0      0        2 eth0
192.168.2.102   vmail-mx.dca.ne .                     0      0        4 eth0

Jul 13 15:15:49 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 13 15:15:49 mindpipe kernel:  [__crc_totalram_pages+1387264/5353478] snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 13 15:15:49 mindpipe kernel:  [__crc_totalram_pages+1455070/5353478] snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 13 15:15:49 mindpipe kernel:  [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 13 15:15:49 mindpipe kernel:  [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 13 15:15:49 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 13 15:15:49 mindpipe kernel:  [rt_run_flush+77/144] rt_run_flush+0x4d/0x90
Jul 13 15:15:49 mindpipe kernel:  [rt_secret_rebuild+0/48] rt_secret_rebuild+0x0/0x30
Jul 13 15:15:49 mindpipe kernel:  [rt_secret_rebuild+14/48] rt_secret_rebuild+0xe/0x30
Jul 13 15:15:49 mindpipe kernel:  [run_timer_softirq+201/416] run_timer_softirq+0xc9/0x1a0
Jul 13 15:15:49 mindpipe kernel:  [__do_softirq+131/144] __do_softirq+0x83/0x90
Jul 13 15:15:49 mindpipe kernel:  [do_softirq+53/64] do_softirq+0x35/0x40
Jul 13 15:15:49 mindpipe kernel:  [do_IRQ+261/352] do_IRQ+0x105/0x160
Jul 13 15:15:49 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 13 15:15:49 mindpipe kernel:  [default_idle+35/64] default_idle+0x23/0x40
Jul 13 15:15:49 mindpipe kernel:  [apm_cpu_idle+165/384] apm_cpu_idle+0xa5/0x180
Jul 13 15:15:49 mindpipe kernel:  [cpu_idle+45/64] cpu_idle+0x2d/0x40
Jul 13 15:15:49 mindpipe kernel:  [start_kernel+392/464] start_kernel+0x188/0x1d0
Jul 13 15:15:49 mindpipe kernel:
Jul 13 15:25:49 mindpipe kernel: ALSA /usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 13 15:25:49 mindpipe kernel:  [__crc_totalram_pages+1387264/5353478] snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 13 15:25:49 mindpipe kernel:  [__crc_totalram_pages+1455070/5353478] snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 13 15:25:49 mindpipe kernel:  [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 13 15:25:49 mindpipe kernel:  [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 13 15:25:49 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 13 15:25:49 mindpipe kernel:  [local_bh_enable+17/112] local_bh_enable+0x11/0x70
Jul 13 15:25:49 mindpipe kernel:  [rt_run_flush+92/144] rt_run_flush+0x5c/0x90
Jul 13 15:25:49 mindpipe kernel:  [rt_secret_rebuild+0/48] rt_secret_rebuild+0x0/0x30
Jul 13 15:25:49 mindpipe kernel:  [rt_secret_rebuild+14/48] rt_secret_rebuild+0xe/0x30
Jul 13 15:25:49 mindpipe kernel:  [run_timer_softirq+201/416] run_timer_softirq+0xc9/0x1a0
Jul 13 15:25:49 mindpipe kernel:  [__do_softirq+131/144] __do_softirq+0x83/0x90
Jul 13 15:25:49 mindpipe kernel:  [do_softirq+53/64] do_softirq+0x35/0x40
Jul 13 15:25:49 mindpipe kernel:  [do_IRQ+261/352] do_IRQ+0x105/0x160
Jul 13 15:25:49 mindpipe kernel:  [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 13 15:25:49 mindpipe kernel:  [default_idle+35/64] default_idle+0x23/0x40
Jul 13 15:25:49 mindpipe kernel:  [apm_cpu_idle+165/384] apm_cpu_idle+0xa5/0x180
Jul 13 15:25:49 mindpipe kernel:  [cpu_idle+45/64] cpu_idle+0x2d/0x40
Jul 13 15:25:49 mindpipe kernel:  [start_kernel+392/464] start_kernel+0x188/0x1d0
Jul 13 15:25:49 mindpipe kernel:

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Bill Davidsen]

Benno Senoner wrote:
> Lee Revell wrote:
> 
>> On Mon, 2004-07-12 at 19:31, Andrew Morton wrote:
>>  
>>
>>>
>>> OK, thanks.  The problem areas there are the timer-based route cache
>>> flushing and reiserfs.
>>>
>>> We can probably fix the route caceh thing by rescheduling the timer 
>>> after
>>> having handled 1000 routes or whatever, although I do wonder if this 
>>> is a
>>> thing we really need to bother about - what else was that machine up to?
>>>
>>>   
>>
>>
>> Gnutella client.  Forgot about that.  I agree, it is not reasonable to
>> expect low latency with this kind of network traffic happening.  I am
>> impressed it worked as well as it did.
>>  
>>
> 
> Why not reasonable ? It is very important that networking and HD I/O 
> both don't interfere with low latency audio.
> Think about large audio setups where you use PC hardware to act as 
> dedicated samplers, software synthesizers etc.
> Such clusters might be diskless and communicate with a GBit ethernet 
> with a high performance file server and
> in some cases lots of network I/O might occur during audio playback. So 
> having latency spikes during network I/O
> would be a big showstopper for many apps in certain setups.
> Even ardour if run on a diskless client would need low latency while 
> doing network I/O because it does lots of disk I/O
> which on the diskless client translate to lots of network I/O.
> Typical use could be educational or research institutions where diskless 
> clients drastically lower the cost of managing large number
> of boxes and allow sharing of resources. See the LTSP project.

Having used "diskless" systems off and on for almost two decades, I 
highly suggest that you are better off with a disk in the node, and use 
that for swap and temp. If you place any value on the time of people, 
this will eliminate a lot of performance (time and bandwidth) issues, 
and usually save on the cost of managing, since you have nothing to 
manage on the node and a lot less network traffic to handle.

I've done it with SunOS, Solaris and Linux.

-- 
    -bill davidsen (davidsen@tmr.com)
"The secret to procrastination is to put things off until the
  last possible moment - but no longer"  -me

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Florin Andrei]

On Sat, 2004-07-10 at 22:25, Andrew Morton wrote:

> What we need to do is to encourage audio testers to use ALSA drivers, to
> enable CONFIG_SND_DEBUG in the kernel build and to set
> /proc/asound/*/*/xrun_debug and to send us the traces which result from
> underruns.

Just out of curiosity:
If i enable CONFIG_SND_DEBUG when compiling the kernel, but do not set
/proc/*/xrun_debug, will that kernel be slower than without
CONFIG_SND_DEBUG turned on?

-- 
Florin Andrei

http://florin.myip.org/

Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch

[Takashi Iwai]

At Thu, 15 Jul 2004 21:50:29 -0700,
Florin Andrei wrote:
> 
> On Sat, 2004-07-10 at 22:25, Andrew Morton wrote:
> 
> > What we need to do is to encourage audio testers to use ALSA drivers, to
> > enable CONFIG_SND_DEBUG in the kernel build and to set
> > /proc/asound/*/*/xrun_debug and to send us the traces which result from
> > underruns.
> 
> Just out of curiosity:
> If i enable CONFIG_SND_DEBUG when compiling the kernel, but do not set
> /proc/*/xrun_debug, will that kernel be slower than without
> CONFIG_SND_DEBUG turned on?

XRUN is checked regardless with or without CONFIG_SND_DEBUG, so there
is no difference in the performance.  It will just print out messages
at each XRUN.

With debug option, some debug checks will be compiled in, but the
impact to perfomance must be ignorable.


Takashi