Re: 2.4.20, .text.lock.swap cpu usage? (ibm x440) [rescued]

Re: 2.4.20, .text.lock.swap cpu usage? (ibm x440) [rescued]

[Brian Tinsley]

>
>
>Which driver are you using for the QLA23xx?
>
The 6.01.00-fo driver from QLogic.

>I've missed the beginning of this thread, but problem here may not 
>have anything to with the number of threads you're runnng, rather it 
>may well be the qla2300 driver, if you are using the qlogic v6 driver.
>
I was getting nailed by an issue involving reaping inodes (or the lack 
thereof). I don't believe the QLogic driver was contributing to my 
problem. I've had this driver running in my lab and at numerous client 
sites for quite some time and have never seen it even burp.

>I have a test system in work which is similar to yours (2x qla2310F,
>SMP (dual athlon) attached to FC RAID storage) and it is easy to live
>lock with any kind of intensive IO, eg bonnie++.
>
>The Redhat 5.31-RH driver is about the most stable one, but i havnt
>extensively stress tested it. All of the qlogic v6 drivers are trivial
>to lock. (they spin forever in the qla2300 ISR - qlogic have beta
>drivers, but they still have the same problem).
>
Interesting. Again, I've never seen this behavior, but I appreciate your 
mentioning it. It's definitely something to keep an eye out for.

-- 

Brian Tinsley
Chief Systems Engineer
Emageon

qla2300 driver stability, (was Re: 2.4.20, .text.lock.swap cpu usage?)

[Paul Jakma]

On Sun, 12 Jan 2003, Brian Tinsley wrote:

> The 6.01.00-fo driver from QLogic.

hmm..

> problem. I've had this driver running in my lab and at numerous client 
> sites for quite some time and have never seen it even burp.

how hard did you push it in testing?

in some configurations i've had to subject it to many hours of
intensive IO (ie multiple concurrent and continious bonnie++ runs of
varying file sizes) in order to get it to spin in
qla2x00_intr_handler. but it will eventually hang given enough IO ime.  
(in other configurations, heavy sustained IO will lock it up in
minutes, even seconds).

> Interesting. Again, I've never seen this behavior, but I appreciate
> your mentioning it. It's definitely something to keep an eye out
> for.

regards,
-- 
Paul Jakma	Sys Admin	Alphyra
	paulj@alphyra.ie
Warning: /never/ send email to spam@dishone.st or trap@dishone.st

Re: 2.4.20, .text.lock.swap cpu usage? (ibm x440) [rescued]

[Brian Tinsley]

>
>Okay, can you try with either 2.4.x-aa or 2.5.x-CURRENT?
>
Yes, I *just* booted a machine with 2.4.20-aa1 in our lab. I was having 
problems compiling the Linux Virtual Server code, but it's fixed now. 

>I'm suspecting either bh problems or lowpte problems.
>
>Also, could you monitor your load with the scripts I posted?
>  
>
Yes, they are already uploaded to a customer site and ready to go. I 
need to flex the -aa1 kernel a bit before I load it there as well.


Thanks!


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: 2.4.20, .text.lock.swap cpu usage? (ibm x440) [rescued]

[Martin J. Bligh]

> Pentium 4 Xeon MP processors
> 
> 2 processor system has 4GB RAM
> 4 processor system has 8GB RAM
> 
> 1 IBM ServeRAID controller
> 2 Intel PRO/1000MT NICs
> 2 QLogic 2340 Fibre Channel HBAs
> 
>> Or perhaps the kernel version is not up-to-date. Please also provide
>> the precise kernel version (and included patches). And workload too.
>> 
> The kernel version is stock 2.4.20 with Chris Mason's data logging and journal relocation patches for ReiserFS (neither of which are actually in use for any mounted filesystems). It is compiled for 64GB highmem support. And just to refresh, I have seen this exact behavior on stock 2.4.19 and stock 2.4.17 (no patches on either of these) also compiled with 64GB highmem support.
> 
> Workload:
> When the live-lock occurs, the system is performing intensive network I/O and intensive disk reads from the fibre channel storage (i.e., the backup program is reading files from disk and transferring them to the backup server). I posted a snapshot of sar data collection earlier today showing selected stats leading up to and just after the live-lock occurs (which is noted by a ~2 minute gap in sar logging). After the live-lock is released, the only thing that stands out is an unusual increase in runtime for kswapd (as reported by ps).
> 
> The various Java programs mentioned in prior postings are *mostly* idle at this point in time as it is after hours for our clients.


If you don't have any individual processes that need to be particularly
large (eg > 1Gb of data), I suggest you just cheat^Wfinesse the problem
and move PAGE_OFFSET from C0000000 to 80000000 - will give you more than
twice as much lowmem to play with. I think this might even be a config
option in RedHat kernels.

Martin.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: 2.4.20, .text.lock.swap cpu usage? (ibm x440) [rescued]

[Brian Tinsley]

>
>We're straying from the subject here.
>
Sorry

>Please describe your machine,
>in terms of how many cpus it has and how much highmem it has, and
>your workload, so I can better determine the issue. Perhaps we can
>cooperatively devise something that works well for you.
>
IBM x360
Pentium 4 Xeon MP processors

2 processor system has 4GB RAM
4 processor system has 8GB RAM

1 IBM ServeRAID controller
2 Intel PRO/1000MT NICs
2 QLogic 2340 Fibre Channel HBAs

>Or perhaps the kernel version is not up-to-date. Please also provide
>the precise kernel version (and included patches). And workload too.
>
The kernel version is stock 2.4.20 with Chris Mason's data logging and 
journal relocation patches for ReiserFS (neither of which are actually 
in use for any mounted filesystems). It is compiled for 64GB highmem 
support. And just to refresh, I have seen this exact behavior on stock 
2.4.19 and stock 2.4.17 (no patches on either of these) also compiled 
with 64GB highmem support.

Workload:
When the live-lock occurs, the system is performing intensive network 
I/O and intensive disk reads from the fibre channel storage (i.e., the 
backup program is reading files from disk and transferring them to the 
backup server). I posted a snapshot of sar data collection earlier today 
showing selected stats leading up to and just after the live-lock occurs 
(which is noted by a ~2 minute gap in sar logging). After the live-lock 
is released, the only thing that stands out is an unusual increase in 
runtime for kswapd (as reported by ps).

The various Java programs mentioned in prior postings are *mostly* idle 
at this point in time as it is after hours for our clients.


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: 2.4.20, .text.lock.swap cpu usage? (ibm x440) [rescued]

[Brian Tinsley]

>William Lee Irwin III wrote:
>  
>
>>>IMHO multiprogramming is as valid a use for memory as any other. Or
>>>even otherwise, it's not something I care to get in design debates
>>>about, it's just how the things are used.
>>>      
>>>
>
>On Thu, Jan 09, 2003 at 09:42:06PM -0600, Brian Tinsley wrote:
>  
>
>>I agree with the philosophy in general, but if I sit down to write a 
>>threaded application for Linux on IA-32 and wind up with a design that 
>>uses 800+ threads in any instance (other than a bug, which was our 
>>case), it's time to give up the day job and start riding on the back of 
>>the garbage truck ;)
>>    
>>
>
>I could care less what userspace does: mechanism, not policy. Userspace
>wants, and I give if I can, just as the kernel does with system calls.
>
>800 threads isn't even a high thread count anyway, the 2.5.x testing
>was with a peak thread count of 100,000. 800 threads, even with an 8KB
>stack, is no more than 6.4MB of lowmem for stacks and so shouldn't
>stress the system unless many instances of it are run.
>
I understand your perspective here. I won't get into application design 
issues as it is far out of context from this list.

>I suspect your issue is elsewhere. I'll submit accounting patches for Marcelo's and/or Andrea's trees so you can find out what's actually going on.
>
Much appreciated! I look forward to it.


>On Thu, Jan 09, 2003 at 09:42:06PM -0600, Brian Tinsley wrote:
>  
>
>>In all honesty, I would enjoy nothing more than contributing to kernel 
>>development. Unfortunately it's a bit out of my scope right now (but not forever). If I only believed aliens seeded our gene pool with clones, I could hook up with those folks that claim to have cloned a human and get one of me made! ;)
>>    
>>
>
>I don't know what to tell you here. I'm lucky that this is my day job
>and that I can contribute so much. However, there are plenty who
>contribute major changes (many even more important than my own) without
>any such sponsorship. Perhaps emulating them would satisfy your wish.
>
It would!

I cannot say thanks enough for the efforts of you and everyone else out 
there. Frankly, I would not have my day job and would not have been able 
to make Emageon what it is today were it not for you all!

Oh, please excuse the stupid humor tonight. I'm in a giddy mood for some 
reason. Must be the excitement from the prospect of getting resolution 
to this problem!


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: 2.4.20, .text.lock.swap cpu usage? (ibm x440) [rescued]

[William Lee Irwin III]

>> IMHO multiprogramming is as valid a use for memory as any other. Or
>> even otherwise, it's not something I care to get in design debates
>> about, it's just how the things are used.

On Thu, Jan 09, 2003 at 09:42:06PM -0600, Brian Tinsley wrote:
> I agree with the philosophy in general, but if I sit down to write a 
> threaded application for Linux on IA-32 and wind up with a design that 
> uses 800+ threads in any instance (other than a bug, which was our 
> case), it's time to give up the day job and start riding on the back of 
> the garbage truck ;)

I could care less what userspace does: mechanism, not policy. Userspace
wants, and I give if I can, just as the kernel does with system calls.

800 threads isn't even a high thread count anyway, the 2.5.x testing
was with a peak thread count of 100,000. 800 threads, even with an 8KB
stack, is no more than 6.4MB of lowmem for stacks and so shouldn't
stress the system unless many instances of it are run. I suspect your
issue is elsewhere. I'll submit accounting patches for Marcelo's and/or
Andrea's trees so you can find out what's actually going on.


William Lee Irwin III wrote:
>> Only you, if anyone. My intentions and patchwriting efforts on the 64GB
>> and highmem multiprogramming fronts are long since public, and publicly
>> stated to be targeted at 2.7. Since there isn't a 2.7 yet, 2.5-CURRENT
>> must suffice until there is.

On Thu, Jan 09, 2003 at 09:42:06PM -0600, Brian Tinsley wrote:
> In all honesty, I would enjoy nothing more than contributing to kernel 
> development. Unfortunately it's a bit out of my scope right now (but not 
> forever). If I only believed aliens seeded our gene pool with clones, I 
> could hook up with those folks that claim to have cloned a human and get 
> one of me made! ;)

I don't know what to tell you here. I'm lucky that this is my day job
and that I can contribute so much. However, there are plenty who
contribute major changes (many even more important than my own) without
any such sponsorship. Perhaps emulating them would satisfy your wish.


Bill
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: 2.4.20, .text.lock.swap cpu usage? (ibm x440) [rescued]

[William Lee Irwin III]

>> Or perhaps the kernel version is not up-to-date. Please also provide
>> the precise kernel version (and included patches). And workload too.

On Thu, Jan 09, 2003 at 10:50:03PM -0600, Brian Tinsley wrote:
> The kernel version is stock 2.4.20 with Chris Mason's data logging and 
> journal relocation patches for ReiserFS (neither of which are actually 
> in use for any mounted filesystems). It is compiled for 64GB highmem 
> support. And just to refresh, I have seen this exact behavior on stock 
> 2.4.19 and stock 2.4.17 (no patches on either of these) also compiled 
> with 64GB highmem support.

Okay, can you try with either 2.4.x-aa or 2.5.x-CURRENT?

I'm suspecting either bh problems or lowpte problems.

Also, could you monitor your load with the scripts I posted?


Thanks,
Bill
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/