Re: Serial port latency

Re: Serial port latency

[Pavel Machek]

Hi!

> Is the computer otherwise idle?
> I've seen one unexplainable report with atm problems that disappeared
> (!) if a kernel compile was running.

I've seen similar bugs. If you hook something on schedule_tq and forget
to set current->need_resched, this is exactly what you get.
-- 
Philips Velo 1: 1"x4"x8", 300gram, 60, 12MB, 40bogomips, linux, mutt,
details at http://atrey.karlin.mff.cuni.cz/~pavel/velo/index.html.

Re: Serial port latency

[Manfred Spraul]

From: "Pavel Machek" <pavel@suse.cz>
> > Is the computer otherwise idle?
> > I've seen one unexplainable report with atm problems that
disappeared
> > (!) if a kernel compile was running.
>
> I've seen similar bugs. If you hook something on schedule_tq and
forget
> to set current->need_resched, this is exactly what you get.
>
I'm running with a patch that printk's if cpu_idle() is called while a
softirq is pending.
If I access the floppy on my K6/200 every track triggers the check, and
sometimes the console blanking code triggers it.

What about creating a special cpu_is_idle() function that the idle
functions must call before sleeping?

--
    Manfred

Re: Serial port latency

[Pavel Machek]

Hi!

> > > Is the computer otherwise idle?
> > > I've seen one unexplainable report with atm problems that
> disappeared
> > > (!) if a kernel compile was running.
> >
> > I've seen similar bugs. If you hook something on schedule_tq and
> forget
> > to set current->need_resched, this is exactly what you get.
> >
> I'm running with a patch that printk's if cpu_idle() is called while a
> softirq is pending.
> If I access the floppy on my K6/200 every track triggers the check, and
> sometimes the console blanking code triggers it.

Seems floppy and console is buggy, then.

> What about creating a special cpu_is_idle() function that the idle
> functions must call before sleeping?

I'd say just fix all the bugs.
								Pavel
-- 
The best software in life is free (not shareware)!		Pavel
GCM d? s-: !g p?:+ au- a--@ w+ v- C++@ UL+++ L++ N++ E++ W--- M- Y- R+

Re: Serial port latency

[Manfred Spraul]

[-- Attachment #1: Type: text/plain, Size: 1514 bytes --]

Pavel Machek wrote:
> 
> > >
> > > I've seen similar bugs. If you hook something on schedule_tq and
> > forget
> > > to set current->need_resched, this is exactly what you get.
> > >
> > I'm running with a patch that printk's if cpu_idle() is called while a
> > softirq is pending.
> > If I access the floppy on my K6/200 every track triggers the check, and
> > sometimes the console blanking code triggers it.
> 
> Seems floppy and console is buggy, then.
>

No. The softirq implementation is buggy.
I can trigger the problem with the TASKLET_HI (floppy), and both net rx
and tx (ping -l)

> > What about creating a special cpu_is_idle() function that the idle
> > functions must call before sleeping?
> 
> I'd say just fix all the bugs.
>

Ok, there are 2 bugs that are (afaics) impossible to fix without
checking for pending softirq's in cpu_idle():

a)
queue_task(my_task1, tq_immediate);
mark_bh();
schedule();
;within schedule: do_softirq()
;within my_task1:
mark_bh();
; bh returns, but do_softirq won't loop
; do_softirq returns.
; schedule() clears current->need_resched
; idle thread scheduled.
--> idle can run although softirq's are pending

I assume I trigger this race with the floppy driver.

b)
hw interrupt
do_softirq
within the net_rx handler: another hw interrupt, additional packets are
queued
do_softirq won't loop.
returns to idle thread. --> packets delayed unnecessary.

What about the attached patch? Obviously the other idle cpu must be
converted to use the function as well.

--
	Manfred

[-- Attachment #2: patch-proc --]
[-- Type: text/plain, Size: 1025 bytes --]

--- 2.4/arch/i386/kernel/process.c	Thu Feb 22 22:28:52 2001
+++ build-2.4/arch/i386/kernel/process.c	Sun Apr  1 00:05:21 2001
@@ -73,6 +73,30 @@
 	hlt_counter--;
 }
 
+/**
+ * cpu_is_idle - helper function for idle functions
+ * 
+ * pm_idle functions must call this function to verify that
+ * the cpu is really idle. It must be called with disabled
+ * local interrupts.
+ * Return values:
+ * 0: cpu was not idle, local interrupts reenabled.
+ * 1: go into power saving mode, local interrupts are
+ *    still disabled.
+*/
+static inline int cpu_is_idle(void)
+{
+	if (current->need_resched) {
+		__sti();
+		return 0;
+	}
+	if (softirq_active(smp_processor_id()) & softirq_mask(smp_processor_id())) {
+		__sti();
+		do_softirq();
+		return 0;
+	}
+	return 1;
+}
 /*
  * We use this if we don't have any better
  * idle routine..
@@ -81,10 +105,8 @@
 {
 	if (current_cpu_data.hlt_works_ok && !hlt_counter) {
 		__cli();
-		if (!current->need_resched)
+		if (cpu_is_idle())
 			safe_halt();
-		else
-			__sti();
 	}
 }
 

softirq buggy [Re: Serial port latency]

[Pavel Machek]

Hi!

> > Seems floppy and console is buggy, then.
> >
> 
> No. The softirq implementation is buggy.
> I can trigger the problem with the TASKLET_HI (floppy), and both net rx
> and tx (ping -l)
> 
> > > What about creating a special cpu_is_idle() function that the idle
> > > functions must call before sleeping?
> > 
> > I'd say just fix all the bugs.
> >
> 
> Ok, there are 2 bugs that are (afaics) impossible to fix without
> checking for pending softirq's in cpu_idle():
> 
> a)
> queue_task(my_task1, tq_immediate);
> mark_bh();
> schedule();
> ;within schedule: do_softirq()
> ;within my_task1:
> mark_bh();
> ; bh returns, but do_softirq won't loop
> ; do_softirq returns.
> ; schedule() clears current->need_resched
> ; idle thread scheduled.
> --> idle can run although softirq's are pending

Or anything else can run altrough softirqs are pending. If it is
computation job, softinterrupts are delayed quiet a bit, right?

So right fix seems to be "loop in do_softirq".

								Pavel

> I assume I trigger this race with the floppy driver.
> 
> b)
> hw interrupt
> do_softirq
> within the net_rx handler: another hw interrupt, additional packets are
> queued
> do_softirq won't loop.
> returns to idle thread. --> packets delayed unnecessary.
> 
> What about the attached patch? Obviously the other idle cpu must be
> converted to use the function as well.

-- 
I'm pavel@ucw.cz. "In my country we have almost anarchy and I don't care."
Panos Katsaloulis describing me w.r.t. patents at discuss@linmodems.org

Re: softirq buggy [Re: Serial port latency]

[Manfred Spraul]

From: "Pavel Machek" <pavel@ucw.cz>
>
> > Ok, there are 2 bugs that are (afaics) impossible to fix without
> > checking for pending softirq's in cpu_idle():
> >
> > a)
> > queue_task(my_task1, tq_immediate);
> > mark_bh();
> > schedule();
> > ;within schedule: do_softirq()
> > ;within my_task1:
> > mark_bh();
> > ; bh returns, but do_softirq won't loop
> > ; do_softirq returns.
> > ; schedule() clears current->need_resched
> > ; idle thread scheduled.
> > --> idle can run although softirq's are pending
>
> Or anything else can run altrough softirqs are pending. If it is
> computation job, softinterrupts are delayed quiet a bit, right?
>
> So right fix seems to be "loop in do_softirq".
>
No, it's the wrong fix.
A network server under high load would loop forever within the softirq,
never returning to process level.

do_softirq cannot loop, the right fix is "check often for pending
softirq's".
It's checked before a process returns to user space, it's checked when a
process schedules. What's missing is that the idle functions must check
for pending softirqs, too.

--
    Manfred

Re: softirq buggy [Re: Serial port latency]

[Pavel Machek]

Hi!

> > > Ok, there are 2 bugs that are (afaics) impossible to fix without
> > > checking for pending softirq's in cpu_idle():
> > >
> > > a)
> > > queue_task(my_task1, tq_immediate);
> > > mark_bh();
> > > schedule();
> > > ;within schedule: do_softirq()
> > > ;within my_task1:
> > > mark_bh();
> > > ; bh returns, but do_softirq won't loop
> > > ; do_softirq returns.
> > > ; schedule() clears current->need_resched
> > > ; idle thread scheduled.
> > > --> idle can run although softirq's are pending
> >
> > Or anything else can run altrough softirqs are pending. If it is
> > computation job, softinterrupts are delayed quiet a bit, right?
> >
> > So right fix seems to be "loop in do_softirq".
> >
> No, it's the wrong fix.
> A network server under high load would loop forever within the softirq,
> never returning to process level.
> 
> do_softirq cannot loop, the right fix is "check often for pending
> softirq's".
> It's checked before a process returns to user space, it's checked when a
> process schedules. What's missing is that the idle functions must check
> for pending softirqs, too.

Ok. I was missing the fact it is checked on going userspace.

-- 
The best software in life is free (not shareware)!		Pavel
GCM d? s-: !g p?:+ au- a--@ w+ v- C++@ UL+++ L++ N++ E++ W--- M- Y- R+

Re: softirq buggy [Re: Serial port latency]

[Manfred Spraul]

[-- Attachment #1: Type: text/plain, Size: 132 bytes --]

Pavel Machek wrote:
> 
> Ok. I was missing the fact it is checked on going userspace.
>

What about the attached patch?

--
	Manfred

[-- Attachment #2: patch-cpu-idle --]
[-- Type: text/plain, Size: 2858 bytes --]

// $Header$
// Kernel Version:
//  VERSION = 2
//  PATCHLEVEL = 4
//  SUBLEVEL = 3
//  EXTRAVERSION = -ac3
--- 2.4/include/linux/interrupt.h	Thu Feb 22 22:29:53 2001
+++ build-2.4/include/linux/interrupt.h	Sat Apr  7 23:51:31 2001
@@ -269,4 +269,25 @@
 extern int probe_irq_off(unsigned long);	/* returns 0 or negative on failure */
 extern unsigned int probe_irq_mask(unsigned long);	/* returns mask of ISA interrupts */
 
+/**
+ * cpu_is_idle - helper function for idle functions
+ * 
+ * pm_idle functions must call this function to verify that
+ * the cpu is really idle. It must be called with disabled
+ * local interrupts.
+ * Return values:
+ * 0: cpu was not idle.
+ * 1: go into power saving mode.
+ */
+static inline int cpu_is_idle(void)
+{
+	if (current->need_resched) {
+		return 0;
+	}
+	if (softirq_active(smp_processor_id()) & softirq_mask(smp_processor_id())) {
+		do_softirq();
+		return 0;
+	}
+	return 1;
+}
 #endif
--- 2.4/arch/i386/kernel/process.c	Thu Feb 22 22:28:52 2001
+++ build-2.4/arch/i386/kernel/process.c	Sat Apr  7 23:51:46 2001
@@ -73,6 +73,7 @@
 	hlt_counter--;
 }
 
+
 /*
  * We use this if we don't have any better
  * idle routine..
@@ -81,7 +82,7 @@
 {
 	if (current_cpu_data.hlt_works_ok && !hlt_counter) {
 		__cli();
-		if (!current->need_resched)
+		if (cpu_is_idle())
 			safe_halt();
 		else
 			__sti();
@@ -92,26 +93,21 @@
  * On SMP it's slightly faster (but much more power-consuming!)
  * to poll the ->need_resched flag instead of waiting for the
  * cross-CPU IPI to arrive. Use this option with caution.
+ *
+ * Isn't this identical to default_idle with the 'no-hlt' boot
+ * option? <manfred@colorfullife.com>
  */
 static void poll_idle (void)
 {
 	int oldval;
 
+	for(;;) {
+		if (!cpu_is_idle())
+			break;
+		__sti();
+		rep_nop();
+	}
 	__sti();
-
-	/*
-	 * Deal with another CPU just having chosen a thread to
-	 * run here:
-	 */
-	oldval = xchg(&current->need_resched, -1);
-
-	if (!oldval)
-		asm volatile(
-			"2:"
-			"cmpl $-1, %0;"
-			"rep; nop;"
-			"je 2b;"
-				: :"m" (current->need_resched));
 }
 
 /*
--- 2.4/drivers/acpi/cpu.c	Sat Apr  7 22:02:01 2001
+++ build-2.4/drivers/acpi/cpu.c	Sat Apr  7 23:55:17 2001
@@ -148,7 +148,7 @@
 		unsigned long diff;
 		
 		__cli();
-		if (current->need_resched)
+		if (!cpu_is_idle())
 			goto out;
 		if (acpi_bm_activity())
 			goto sleep2;
@@ -171,7 +171,7 @@
 		unsigned long diff;
 
 		__cli();
-		if (current->need_resched)
+		if (!cpu_is_idle())
 			goto out;
 		if (acpi_bm_activity())
 			goto sleep2;
@@ -205,7 +205,7 @@
 		unsigned long diff;
 
 		__cli();
-		if (current->need_resched)
+		if (!cpu_is_idle())
 			goto out;
 
 		time = acpi_read_pm_timer();
@@ -235,7 +235,7 @@
 		unsigned long diff;
 
 		__cli();
-		if (current->need_resched)
+		if (!cpu_is_idle())
 			goto out;
 		time = acpi_read_pm_timer();
 		acpi_c1_count++;

Re: softirq buggy [Re: Serial port latency]

[kuznet]

Hello!

> +	if (softirq_active(smp_processor_id()) & softirq_mask(smp_processor_id())) {
> +		do_softirq();
> +		return 0;

BTW you may delete do_softirq()... schedule() will call this.


> + *
> + * Isn't this identical to default_idle with the 'no-hlt' boot
> + * option? <manfred@colorfullife.com>

Seeems, it is not. need_resched=-1 avoids useless IPIs.

Alexey

Re: softirq buggy [Re: Serial port latency]

[Manfred Spraul]

From: <kuznet@ms2.inr.ac.ru>
> Hello!
>
> > + if (softirq_active(smp_processor_id()) &
softirq_mask(smp_processor_id())) {
> > + do_softirq();
> > + return 0;
>
> BTW you may delete do_softirq()... schedule() will call this.
>

But with a huge overhead. I'd prefer to call it directly from within the
idle functions, the overhead of schedule is IMHO too high.

>
> > + *
> > + * Isn't this identical to default_idle with the 'no-hlt' boot
> > + * option? <manfred@colorfullife.com>
>
> Seeems, it is not. need_resched=-1 avoids useless IPIs.
>
I already wondered why need_resched is set to -1 ;-)

I'll remove that comment and repost the patch.

--
    Manfred

Re: softirq buggy [Re: Serial port latency]

[kuznet]

Hello!

> But with a huge overhead. I'd prefer to call it directly from within the
> idle functions, the overhead of schedule is IMHO too high.


+	if (current->need_resched) {
+		return 0;
		^^^^^^^^
+	}
+	if (softirq_active(smp_processor_id()) & softirq_mask(smp_processor_id())) {
+		do_softirq();
+		return 0;
		^^^^^^^^^
You return one value in both casesand I decided it means "schedule". 8)
Apparently you meaned return 1 in the first case. 8)

But in this case it becomes wrong. do_softirq() can raise need_reshed
and moreover irqs arrive during it. Order of check should be different.


BTW what's about overhead... I suspect it is _lower_ in the case
of schedule(). In the case of networking at least, when softirq
most likely wakes some socket.

Alexey

Re: softirq buggy [Re: Serial port latency]

[Manfred Spraul]

From: <kuznet@ms2.inr.ac.ru>
To: "Manfred Spraul" <manfred@colorfullife.com>
Cc: <linux-kernel@vger.redhat.com>
Sent: Sunday, April 08, 2001 7:58 PM
Subject: Re: softirq buggy [Re: Serial port latency]


> Hello!
>
> > But with a huge overhead. I'd prefer to call it directly from within
the
> > idle functions, the overhead of schedule is IMHO too high.
>
>
> + if (current->need_resched) {
> + return 0;
> ^^^^^^^^
> + }
> + if (softirq_active(smp_processor_id()) &
softirq_mask(smp_processor_id())) {
> + do_softirq();
> + return 0;
> ^^^^^^^^^
> You return one value in both casesand I decided it means "schedule".
8)
> Apparently you meaned return 1 in the first case. 8)
>
No, the code is correct. 0 means "don't stop the cpu".
The pm_idle function pointer will return to cpu_idle()
(arch/i386/kernel/process.c), and that function contains another

    while(!current->need_resched)
        idle();

loop ;-)

> But in this case it becomes wrong. do_softirq() can raise need_reshed
> and moreover irqs arrive during it. Order of check should be
different.
>
Yes, I'll correct that.

>
> BTW what's about overhead... I suspect it is _lower_ in the case
> of schedule(). In the case of networking at least, when softirq
> most likely wakes some socket.
>
I'm not sure - what if the computer is just a router?
But OTHO: the cpu is idle, so it doesn't matter at all if the idle cpu
spends it's time within schedule() or within safe_hlt(), I'll change my
patch.

I have another question:
I added cpu_is_idle() into <linux/interrupt.h>. Is that acceptable, or
is there a better header file for such a function?

--
    Manfred

[PATCH] Re: softirq buggy

[Manfred Spraul]

[-- Attachment #1: Type: text/plain, Size: 377 bytes --]

I've attached a new patch:

* cpu_is_idle() moved to <linux/pm.h>
* function uninlined due to header dependencies
* cpu_is_idle() doesn't call do_softirq directly, instead the caller
returns to schedule()
* cpu_is_idle() exported for modules.
* docu updated.

I'd prefer to inline cpu_is_idle(), but optimizing the idle code path is
probably not that important ;-)

--
	Manfred

[-- Attachment #2: patch-cpu-idle --]
[-- Type: text/plain, Size: 2940 bytes --]

// $Header$
// Kernel Version:
//  VERSION = 2
//  PATCHLEVEL = 4
//  SUBLEVEL = 3
//  EXTRAVERSION = -ac3
--- 2.4/include/linux/pm.h	Thu Jan  4 23:50:47 2001
+++ build-2.4/include/linux/pm.h	Sun Apr  8 21:02:02 2001
@@ -186,6 +186,8 @@
 extern void (*pm_idle)(void);
 extern void (*pm_power_off)(void);
 
+int cpu_is_idle(void);
+
 #endif /* __KERNEL__ */
 
 #endif /* _LINUX_PM_H */
--- 2.4/kernel/sched.c	Sat Apr  7 22:02:27 2001
+++ build-2.4/kernel/sched.c	Sun Apr  8 21:02:37 2001
@@ -1242,6 +1242,28 @@
 	sched_data->last_schedule = get_cycles();
 }
 
+/**
+ * cpu_is_idle - helper function for idle functions
+ * 
+ * pm_idle functions must call this function to verify that
+ * the cpu is really idle.
+ * The return value is valid until local interrupts are
+ * reenabled.
+ * Return values:
+ * 1: go into power saving mode.
+ * 0: cpu is not idle, return to schedule()
+ */
+int cpu_is_idle(void)
+{
+	if (current->need_resched)
+		return 0;
+
+	if (softirq_active(smp_processor_id()) & softirq_mask(smp_processor_id()))
+		return 0;
+
+	return 1;
+}
+
 extern void init_timervecs (void);
 
 void __init sched_init(void)
--- 2.4/kernel/ksyms.c	Sat Apr  7 22:02:27 2001
+++ build-2.4/kernel/ksyms.c	Sun Apr  8 21:42:48 2001
@@ -440,6 +440,7 @@
 #endif
 EXPORT_SYMBOL(kstat);
 EXPORT_SYMBOL(nr_running);
+EXPORT_SYMBOL(cpu_is_idle);
 
 /* misc */
 EXPORT_SYMBOL(panic);
--- 2.4/arch/i386/kernel/process.c	Thu Feb 22 22:28:52 2001
+++ build-2.4/arch/i386/kernel/process.c	Sun Apr  8 21:25:23 2001
@@ -81,7 +81,7 @@
 {
 	if (current_cpu_data.hlt_works_ok && !hlt_counter) {
 		__cli();
-		if (!current->need_resched)
+		if (cpu_is_idle())
 			safe_halt();
 		else
 			__sti();
@@ -105,13 +105,10 @@
 	 */
 	oldval = xchg(&current->need_resched, -1);
 
-	if (!oldval)
-		asm volatile(
-			"2:"
-			"cmpl $-1, %0;"
-			"rep; nop;"
-			"je 2b;"
-				: :"m" (current->need_resched));
+	if (!oldval) {
+		while(cpu_is_idle())
+			rep_nop();
+	}
 }
 
 /*
@@ -131,7 +128,7 @@
 		void (*idle)(void) = pm_idle;
 		if (!idle)
 			idle = default_idle;
-		while (!current->need_resched)
+		while (cpu_is_idle())
 			idle();
 		schedule();
 		check_pgt_cache();
--- 2.4/drivers/acpi/cpu.c	Sat Apr  7 22:02:01 2001
+++ build-2.4/drivers/acpi/cpu.c	Sat Apr  7 23:55:17 2001
@@ -148,7 +148,7 @@
 		unsigned long diff;
 		
 		__cli();
-		if (current->need_resched)
+		if (!cpu_is_idle())
 			goto out;
 		if (acpi_bm_activity())
 			goto sleep2;
@@ -171,7 +171,7 @@
 		unsigned long diff;
 
 		__cli();
-		if (current->need_resched)
+		if (!cpu_is_idle())
 			goto out;
 		if (acpi_bm_activity())
 			goto sleep2;
@@ -205,7 +205,7 @@
 		unsigned long diff;
 
 		__cli();
-		if (current->need_resched)
+		if (!cpu_is_idle())
 			goto out;
 
 		time = acpi_read_pm_timer();
@@ -235,7 +235,7 @@
 		unsigned long diff;
 
 		__cli();
-		if (current->need_resched)
+		if (!cpu_is_idle())
 			goto out;
 		time = acpi_read_pm_timer();
 		acpi_c1_count++;

Re: [PATCH] Re: softirq buggy

[Albert D. Cahalan]

> I'd prefer to inline cpu_is_idle(), but optimizing the idle
> code path is probably not that important ;-)

Sure it is, in one way: how fast can you get back to work?
(not OK to take a millisecond getting out of the idle loop)

Re: [PATCH] Re: softirq buggy

[Andrea Arcangeli]

On Sun, Apr 08, 2001 at 11:35:36PM +0200, Manfred Spraul wrote:
> I've attached a new patch:
> 
> * cpu_is_idle() moved to <linux/pm.h>
> * function uninlined due to header dependencies
> * cpu_is_idle() doesn't call do_softirq directly, instead the caller
> returns to schedule()
> * cpu_is_idle() exported for modules.
> * docu updated.
> 
> I'd prefer to inline cpu_is_idle(), but optimizing the idle code path is
> probably not that important ;-)

your cpu_is_idle will return 0 in the need_resched != 0 check even if the cpu
is idle (because of the -1 trick for avoiding the SMP-IPI to notify the cpu).

The issue you are addressing is quite londstanding and it is not only related
to the loop with an idle cpu.

This is the way I prefer to fix it:

	ftp://ftp.us.kernel.org/pub/linux/kernel/people/andrea/patches/v2.4/2.4.4pre1/ksoftirqd-1

Andrea

Re: [PATCH] Re: softirq buggy

[Manfred Spraul]

Andrea Arcangeli wrote:
> 
> your cpu_is_idle will return 0 in the need_resched != 0 check even if the cpu
> is idle (because of the -1 trick for avoiding the SMP-IPI to notify the cpu).
>
Fixed.

> The issue you are addressing is quite londstanding and it is not only related
> to the loop with an idle cpu.
> 
> This is the way I prefer to fix it:
> 
>         ftp://ftp.us.kernel.org/pub/linux/kernel/people/andrea/patches/v2.4/2.4.4pre1/ksoftirqd-1
>
The return path to user space checks for pending softirqs. A delay of
1/HZ is only possible if the cpu loops in kernel space without returning
to user space - and the functions that can loop check
'current->need_resched'. That means that either cpu_is_idle() must be
renamed to schedule_required() and all 'need_resched' users should use
that function, or something like your patch.

Is a full thread really necessary? Just setting 'need_resched' should be
enough, schedule() checks for pending softirqs.
And do you have a rough idea how often that new thread is scheduled
under load?

Btw, you don't schedule the ksoftirqd thread if do_softirq() returns
from the 'if(in_interrupt())' check.
I assume that this is the most common case of delayed softirq
processing:

; in process context
spin_lock_bh();
; hw interrupt arrives
; do_softirq returns immediately
spin_unlock_bh();


--
	Manfred

Re: [PATCH] Re: softirq buggy

[Andrea Arcangeli]

[-- Attachment #1: Type: text/plain, Size: 2730 bytes --]

On Mon, Apr 09, 2001 at 05:26:27PM +0200, Manfred Spraul wrote:
> The return path to user space checks for pending softirqs. A delay of

And it breaks the loop too if new softirq events become pending again in
background.

> 1/HZ is only possible if the cpu loops in kernel space without returning
> to user space - and the functions that can loop check

It is also possible when new events are posted and I think it makes
sense to scale the softirq load with the scheduler when it is flooding.

Theoretically one could move the _whole_ softirq load into the ksoftirqd, but
that would increase the latency too much and I think it is better to use it
only as a fallback when we have to giveup but we still would like to keep
processing the softirq load so we let the scheduler to choose if we still can
do that or if we should giveup on the softirq.

> Is a full thread really necessary? Just setting 'need_resched' should be
> enough, schedule() checks for pending softirqs.

If you abuse need_resched then you can starve userspace again, if you are ok
to starve userspace indefinitely then it is more efficient to keep looping
forever into do_softirq as far as new events are posted in background instead
of exiting do_softirq and waiting the scheduler to kickin again.

> And do you have a rough idea how often that new thread is scheduled
> under load?

The scheduling is not as heavy as with tasks, it's a kernel thread
so the tlb isn't touched. However yes it will generate some overhead
with schedule() compared to just waiting the 1/HZ but letting the scheduler to
understand when the softirq should keep running instead of another task is
supposed to be a feature. I run a netpipe run with an alpha SMP as receiver
with the ksoftirqd patch and then without and the numbers didn't changed at all
even if the ksofitrqd was often running (1/2% of the load of the machine).

> Btw, you don't schedule the ksoftirqd thread if do_softirq() returns
> from the 'if(in_interrupt())' check.

That's not necessary and it's intentional, such check will be passed in the
last do_softirq executed before returning to userspace or kernel normal
context, the reason of such check is only to avoid recursing too much on the
stack during nested irqs.

> I assume that this is the most common case of delayed softirq
> processing:
> 
> ; in process context
> spin_lock_bh();
> ; hw interrupt arrives
> ; do_softirq returns immediately
> spin_unlock_bh();

This is yet another case and it's handled before returning to userspace so the
latency should still be pretty small (and there would be no singificant
advantage and almost certainly only a performance drop in waking up ksoftirqd
from the `do_softirq returns immediatly' line).

Andrea

[-- Attachment #2: ksofitrqd.png --]
[-- Type: image/png, Size: 3940 bytes --]

Re: [PATCH] Re: softirq buggy

[kuznet]

Hello!

> Btw, you don't schedule the ksoftirqd thread if do_softirq() returns
> from the 'if(in_interrupt())' check.

ksoftirqd will not be switched to before the first schedule
or ret form syscall, when softirqs will be processed in any case.
So, wake up in this case would be mistake.


> I assume that this is the most common case of delayed softirq
> processing:

softirqs have the same latency warranty as rt threads, so that
this is not a problem at all.

The _real_ problem is softirqs generated from another softirqs:
additonal thread is made _not_ to speed up softirqs, but to _tame_
them (if I understood Andres's explanations correctly).

Alexey

Re: [PATCH] Re: softirq buggy

[Andrea Arcangeli]

On Mon, Apr 09, 2001 at 09:48:02PM +0400, kuznet@ms2.inr.ac.ru wrote:
> Hello!
> 
> > Btw, you don't schedule the ksoftirqd thread if do_softirq() returns
> > from the 'if(in_interrupt())' check.
> 
> ksoftirqd will not be switched to before the first schedule
> or ret form syscall, when softirqs will be processed in any case.
> So, wake up in this case would be mistake.

Agreed.

> The _real_ problem is softirqs generated from another softirqs:
> additonal thread is made _not_ to speed up softirqs, but to _tame_
> them (if I understood Andres's explanations correctly).

Exactly.

Andrea

Re: Serial port latency

[Andrea Arcangeli]

On Sun, Mar 25, 2001 at 11:10:14PM +0000, Pavel Machek wrote:
> I've seen similar bugs. If you hook something on schedule_tq and forget
> to set current->need_resched, this is exactly what you get.

keventd fixes tq_scheduler case (tq_scheduler is dead).

Andrea

Re: Serial port latency

[Manfred Spraul]

Is the computer otherwise idle?
I've seen one unexplainable report with atm problems that disappeared
(!) if a kernel compile was running.

Could you try to run a simple cpu hog (with nice 20)?

<<
main()
{
    for(;;) getpid();
}
<<

I'm aware of one bug that could cause a delay of up to 20 ms (cpu_idle()
doesn't check for pending softirq's before sleeping), but that doesn't
explain your 500 ms delay.

--
    Manfred

Re: Serial port latency

[Geir Thomassen]

Trent Jarvi wrote:
> 
> Hi,
> 
> I'm not on the kernel list.  I just ran across your email while looking at the
> weekly archive.
> 
> I think you want to enable software flow control.
> 
>         tcgetattr( fd, &ttyset );
>         ttyset.c_iflag |= IXOFF;
>         tcsetattr( fd, TCSANOW, &ttyset );
> 

Just tested it, it didn't change anything. The response from the controller
can contain ^S/^Q, so it would be a bad idea anyway ....

> Someone reported that the Java CommAPI driver at http://www.rxtx.org got
> 150-200ms latency with (9600,N,8,1,XON/XOFF).  Beyond that you may have to
> look at something like the realtime support.  I guess 2 ms is normal on
> win98.

Win98 is the problem I am trying to solve with my program ...

> Since you have the scope hooked up you may look at hardware flow control too.
> 
>         tcgetattr( fd, &ttyset );
>         ttyset.c_cflag |= HARDWARE_FLOW_CONTROL;
>         tcsetattr( fd, TCSANOW, &ttyset );

Do you mean CRTSCTS ? HARDWARE_FLOW_CONTROL is not defined in my header files.
I use CLOCAL, which should make the driver ignore modem control lines.

> Let me know if you find anything out.  I'm the maintainer of rxtx and would
> be interested in documenting this for others.

sure ...

> --
> Trent Jarvi
> TrentJarvi@yahoo.com

Thanks anyway

Geir

Re: Serial port latency

[Trent Jarvi]

On Thu, 22 Mar 2001, Geir Thomassen wrote:

> Trent Jarvi wrote:
> > 
> > Hi,
> > 
> > I'm not on the kernel list.  I just ran across your email while looking at the
> > weekly archive.
> > 
> > I think you want to enable software flow control.
> > 
> >         tcgetattr( fd, &ttyset );
> >         ttyset.c_iflag |= IXOFF;
> >         tcsetattr( fd, TCSANOW, &ttyset );
> > 
> 
> Just tested it, it didn't change anything. The response from the controller
> can contain ^S/^Q, so it would be a bad idea anyway ....
> 
> > Someone reported that the Java CommAPI driver at http://www.rxtx.org got
> > 150-200ms latency with (9600,N,8,1,XON/XOFF).  Beyond that you may have to
> > look at something like the realtime support.  I guess 2 ms is normal on
> > win98.
> 
> Win98 is the problem I am trying to solve with my program ...
> 
> > Since you have the scope hooked up you may look at hardware flow control too.
> > 
> >         tcgetattr( fd, &ttyset );
> >         ttyset.c_cflag |= HARDWARE_FLOW_CONTROL;
> >         tcsetattr( fd, TCSANOW, &ttyset );
> 
> Do you mean CRTSCTS ? HARDWARE_FLOW_CONTROL is not defined in my header files.
> I use CLOCAL, which should make the driver ignore modem control lines.
> 

Ya.. My bad.  I use that to support multiple platforms

#ifdef CRTSCTS
#define HARDWARE_FLOW_CONTROL CRTSCTS
#else

Do you happen to have shared interupts enabled in the kernel?

> > Let me know if you find anything out.  I'm the maintainer of rxtx and would
> > be interested in documenting this for others.
> 
> sure ...
> 
> > --
> > Trent Jarvi
> > TrentJarvi@yahoo.com
> 
> Thanks anyway
> 
> Geir
> 

-- 
Trent Jarvi
TrentJarvi@yahoo.com

Serial port latency

[Geir Thomassen]

Hi. 

My program controls a device (a programmer for microcontrollers) via the
serial port. The program sits in a tight loop, writing a few (typical 6)
bytes to the port, and waits for a few (typ. two) bytes to be returned from
the programmer. 

The program works, but it is very slow. I use an oscilloscope to monitor the
serial lines, and notices that there is a large delay between the returned
data, and the next new command. I really don't know if the delay is on the
sending or the receiving side (or both).

This is what the program does:

     fd=open("/dev/ttyS0",O_NOCTTY | O_RDWR);

     tcsetattr(fd,TCSANOW, &tio); /* setting baud, parity, raw mode, etc */

     while() {
             write( 6 bytes);   /* send command */
             read( 2 bytes);    /* wait for reply */
     }


The device on the serial port responds in typ. 10 ms, but the software uses
over 500ms to get the reply and send the next command. Why is this happening ?
I have a feeling that there is something obvious I am missing (like line
buffering, but that's a stdio (libc) thing, isn't it ?).

BTW, I am running a 2.2 kernel ....

Geir, geirt@powertech.no

Re: Serial port latency

[Theodore Tso]

On Thu, Mar 22, 2001 at 07:21:28PM +0100, Geir Thomassen wrote:
> My program controls a device (a programmer for microcontrollers) via the
> serial port. The program sits in a tight loop, writing a few (typical 6)
> bytes to the port, and waits for a few (typ. two) bytes to be returned from
> the programmer. 

Check out the man page for the "low_latency" configuration parameter
in the setserial man page.  This will cause the serial driver to burn
a small amount of additional CPU overhead when processing characters,
but it will lower the time between when characters arrive at the
RS-232 port and when they are made available to the user program.  The
preferable solution is to use a intelligent windowing protocol that
isn't heavily latency dependent (all modern protocols, such as kermit,
zmodem, tcp/ip, etc. do this).  But if you can't, using setserial to
set the "low_latency" flag will allow you to work around a dumb
communications protocol.

						- Ted

Re: Serial port latency

[Geir Thomassen]

Theodore Tso wrote:
> 
> On Thu, Mar 22, 2001 at 07:21:28PM +0100, Geir Thomassen wrote:
> > My program controls a device (a programmer for microcontrollers) via the
> > serial port. The program sits in a tight loop, writing a few (typical 6)
> > bytes to the port, and waits for a few (typ. two) bytes to be returned from
> > the programmer.
> 
> Check out the man page for the "low_latency" configuration parameter
> in the setserial man page.  This will cause the serial driver to burn
> a small amount of additional CPU overhead when processing characters,
> but it will lower the time between when characters arrive at the
> RS-232 port and when they are made available to the user program.  The
> preferable solution is to use a intelligent windowing protocol that
> isn't heavily latency dependent (all modern protocols, such as kermit,
> zmodem, tcp/ip, etc. do this).  But if you can't, using setserial to
> set the "low_latency" flag will allow you to work around a dumb
> communications protocol.
> 

I have tried this on both a stock Redhat 7.0 kernel and on 2.2.19pre5 with
your serial-5.05 driver, and I could not measure any difference with and
without the "low_latency" parameter to setserial ....

#setserial -a /dev/ttyS1
/dev/ttyS1, Line 1, UART: 16550A, Port: 0x02f8, IRQ: 3
	Baud_base: 115200, close_delay: 50, divisor: 0
	closing_wait: 3000
	Flags: spd_normal skip_test low_latency


The serial port chip is 16550A, which has a built in fifo. Can this be
the source of my problems ?

Geir

Re: Serial port latency

[Paul Fulghum]

> The serial port chip is 16550A, which has a built in fifo. Can this be
> the source of my problems ?
> 
> Geir

I thought about that. If the number of receive bytes in the RX FIFO
is less that the trigger level then a timeout has to occur before
getting the next receive data interrupt.

The 16550AF data book says that this timeout is 4 characters
from when the last byte is received. This is a maximum of 160ms
at 300bps (when using 12bit characters: 1 start + 8 data + parity + 2 stop).

So this would be smaller at 9600 and could not account
for the 500ms delay.

Paul Fulghum paulkf@microgate.com
Microgate Corporation www.microgate.com

Re: Serial port latency

[Theodore Tso]

On Thu, Mar 22, 2001 at 09:32:39PM +0100, Geir Thomassen wrote:
> 
> The serial port chip is 16550A, which has a built in fifo. Can this be
> the source of my problems ?

Well, if you set the uart to be 16450 using setserial, this will cause
Linux to avoid enabling the FIFO.  That will cause the loop to save
the 4 character times (which at 9600 bps is 4ms).  If your original
protocol is writing six characters, and then reads 2 characters in a
tight loop, that means a total cycle takes 8ms, and disabling the FIFO
will have significant savings assuming that all other causes of
latencies have been removed.  (The FIFO delay can cause a slowdown by
50%).

						- Ted

Re: Serial port latency

[Geir Thomassen]

Jim Penny wrote:
> 
> On Thu, Mar 22, 2001 at 07:21:28PM +0100, Geir Thomassen wrote:
> > My program controls a device (a programmer for microcontrollers) via the
> > serial port. The program sits in a tight loop, writing a few (typical 6)
> > bytes to the port, and waits for a few (typ. two) bytes to be returned from
> > the programmer.
> >
> > The program works, but it is very slow. I use an oscilloscope to monitor the
> > serial lines, and notices that there is a large delay between the returned
> > data, and the next new command. I really don't know if the delay is on the
> > sending or the receiving side (or both).
> >


> 
> 500ms seems a bit too long, but the quick answer is scheduler latency.
> You will certainly be scheduled out by the read, after all, it is not
> your program that is doing the read, but the kernel.  I think you are
> scheduled out after the write, as well.

I do believe the write will make the kernel write the first byte
into the UART transmit buffer directly. If this is the case, then the
data should appear on the serial lines almost immediately. There might be a
delay before the bottom half handler process the byte (or is the bottom half
only used on receive ?), but I do believe that this happens before the next
process is scheduled (BH handlers are all finished before any syscalls returns).
If this is the case, then scheduling out on write should not do any harm.

> 
> Your task is then sitting idle until the scheduler's next pass, if nothing
> else is happening, an interrupt is scheduled on the RTC, the CPU is set to
> idle power, and your system sleeps until the RTC wakes it back up, and
> the schedule queue is re-examined.
> 
> This normally implies a wait of at least a jiffy after every read.
>


Isn't a jiffy 10ms on i386 ? Scheduling delays should not be any
more than say 50 ms, but I am measuring 500ms. I could live with
50ms, which would make my program finish in 3 min. Now it is using
30 min !!

> 
> (But your time seems really excessive.)

Yep  :-(

> The usual cure is to write a custom driver which does a single
> read/write command.  If, as I recall, you are scheduled out on reads
> as well as writes, this lets you get rid of half the jiffies.

That could be the solution to my problem, but I won't start on that
before I understand what the problem is ...


Geir, geirt@powertech.no

Re: Serial port latency

[Jonathan Lundell]

>This is what the program does:
>
>     fd=open("/dev/ttyS0",O_NOCTTY | O_RDWR);
>
>     tcsetattr(fd,TCSANOW, &tio); /* setting baud, parity, raw mode, etc */
>
>     while() {
>             write( 6 bytes);   /* send command */
>             read( 2 bytes);    /* wait for reply */
>     }

What are your settings for VTIME and VMIN?

-- 
/Jonathan Lundell.

Re: Serial port latency

[Pavel Machek]

Hi!

> My program controls a device (a programmer for microcontrollers) via the
> serial port. The program sits in a tight loop, writing a few (typical 6)
> bytes to the port, and waits for a few (typ. two) bytes to be returned from
> the programmer. 
> 
> The program works, but it is very slow. I use an oscilloscope to monitor the
> serial lines, and notices that there is a large delay between the returned
> data, and the next new command. I really don't know if the delay is on the
> sending or the receiving side (or both).
> 
> This is what the program does:
> 
>      fd=open("/dev/ttyS0",O_NOCTTY | O_RDWR);
> 
>      tcsetattr(fd,TCSANOW, &tio); /* setting baud, parity, raw mode, etc */
> 
>      while() {
>              write( 6 bytes);   /* send command */
>              read( 2 bytes);    /* wait for reply */
>      }
> 
> 
> The device on the serial port responds in typ. 10 ms, but the software uses
> over 500ms to get the reply and send the next command. Why is this happening ?
> I have a feeling that there is something obvious I am missing (like line
> buffering, but that's a stdio (libc) thing, isn't it ?).

Set HZ=1000 in include/asm/params.h and see if it helps.

								Pavel
-- 
I'm pavel@ucw.cz. "In my country we have almost anarchy and I don't care."
Panos Katsaloulis describing me w.r.t. patents at discuss@linmodems.org

Re: Serial port latency

[Geir Thomassen]

Problem solved !

The main problem was that I didn't know that the equipment used
DTR as character acknowledge (not RTS/CTS which is common for flow
control). I didn't even bother to look at those lines with the
oscilloscope. Now the software runs fine with negligible latency.

Now the total latency is so low that the low_latency option to
setserial really matters, it reduces the total run time of my
program with 38 %

I don't get any measurable speed improvement by forcing the UART
to be detected as a 8250 or a 16450 though.

I have not tested with HZ=1000.

All in all, my problem is solved, thank you all.

Geir