Re: qla2xxx BUG: workqueue leaked lock or atomic

Re: qla2xxx BUG: workqueue leaked lock or atomic

[Andrew Morton]

On Wed, 7 Mar 2007 18:09:55 +0100 Andre Noll <maan@systemlinux.org> wrote:

> On 20:39, Andrew Morton wrote:
> > On Wed, 28 Feb 2007 16:37:22 +0100 Andre Noll <maan@systemlinux.org> wrote:
> > 
> > > On 16:18, Andre Noll wrote:
> > > 
> > > > With 2.6.21-rc2 I am unable to reproduce this BUG message. However,
> > > > writing to both raid systems at the same time via lvm still locks up
> > > > the system within minutes.
> > > 
> > > Screenshot of the resulting kernel panic:
> > > 
> > > 	http://systemlinux.org/~maan/shots/kernel-panic-21-rc2-huangho2.png
> > > 
> > 
> > It died in CFQ.  Please try a different IO scheduler.  Use something
> > like
> > 
> > 	echo deadline > /sys/block/sda/queue/scheduler
> > 
> > This could still be the old qla2xxx bug, or it could be a new qla2xxx bug,
> > or it could be a block bug, or it could be an LVM bug.
> 
> OK. I'm running with deadline right now. But I guess this kernel
> panic was caused by an LVM bug because lockdep reported problems with
> LVM. Nobody responded to my bug report on the LVM mailing list (see
> http://www.redhat.com/archives/linux-lvm/2007-February/msg00102.html).
> 
> Non-working snapshots and no help from the mailing list convinced me
> to ditch the lvm setup [1] in favour of linear software raid. This
> means I can't do lvm-related tests any more.

Sigh.

> BTW: Are ext3 filesystem sizes greater than 8T now officially
> supported?

I think so, but I don't know how much 16TB testing developers and
distros are doing - perhaps the linux-ext4 denizens can tell us?

Re: qla2xxx BUG: workqueue leaked lock or atomic

[Mingming Cao]

On Wed, 2007-03-07 at 11:45 -0800, Andrew Morton wrote:
> On Wed, 7 Mar 2007 18:09:55 +0100 Andre Noll <maan@systemlinux.org> wrote:
> 
> > On 20:39, Andrew Morton wrote:
> > > On Wed, 28 Feb 2007 16:37:22 +0100 Andre Noll <maan@systemlinux.org> wrote:
> > > 
> > > > On 16:18, Andre Noll wrote:
> > > > 
> > > > > With 2.6.21-rc2 I am unable to reproduce this BUG message. However,
> > > > > writing to both raid systems at the same time via lvm still locks up
> > > > > the system within minutes.
> > > > 
> > > > Screenshot of the resulting kernel panic:
> > > > 
> > > > 	http://systemlinux.org/~maan/shots/kernel-panic-21-rc2-huangho2.png
> > > > 
> > > 
> > > It died in CFQ.  Please try a different IO scheduler.  Use something
> > > like
> > > 
> > > 	echo deadline > /sys/block/sda/queue/scheduler
> > > 
> > > This could still be the old qla2xxx bug, or it could be a new qla2xxx bug,
> > > or it could be a block bug, or it could be an LVM bug.
> > 
> > OK. I'm running with deadline right now. But I guess this kernel
> > panic was caused by an LVM bug because lockdep reported problems with
> > LVM. Nobody responded to my bug report on the LVM mailing list (see
> > http://www.redhat.com/archives/linux-lvm/2007-February/msg00102.html).
> > 
> > Non-working snapshots and no help from the mailing list convinced me
> > to ditch the lvm setup [1] in favour of linear software raid. This
> > means I can't do lvm-related tests any more.
> 
> Sigh.
> 
> > BTW: Are ext3 filesystem sizes greater than 8T now officially
> > supported?
> 
> I think so, but I don't know how much 16TB testing developers and
> distros are doing - perhaps the linux-ext4 denizens can tell us?
> -

IBM has done some testing (dbench, fsstress, fsx, tiobench, iozone etc)
on 10TB ext3, I think RedHat and BULL have done similar test on >8TB
ext3 too.

Mingming

Re: qla2xxx BUG: workqueue leaked lock or atomic

[Andre Noll]

[-- Attachment #1: Type: text/plain, Size: 1347 bytes --]

On 12:05, Mingming Cao wrote:
> > > BTW: Are ext3 filesystem sizes greater than 8T now officially
> > > supported?
> > 
> > I think so, but I don't know how much 16TB testing developers and
> > distros are doing - perhaps the linux-ext4 denizens can tell us?
> > -
> 
> IBM has done some testing (dbench, fsstress, fsx, tiobench, iozone etc)
> on 10TB ext3, I think RedHat and BULL have done similar test on >8TB
> ext3 too.

Thanks. I'm asking because some days ago I tried to create a 10T ext3
filesytem on a linear software raid over two hardware raids, and it
failed horribly. mke2fs from e2fsprogs-1.39 refused to create such a
large filesystem but did it with -F, and I could mount it afterwards.
But writing data immediately produced zillions of errors and only
power-cycling the box helped.

We're now using a 7.9T filesystem on the same hardware. That seems
to work fine on 2.6.21-rc2, so I think this is an ext3 problem. I
cannot completely rule out other reasons though as the underlying
qla2xxx driver also had some problems on earlier kernels.

We'd much rather have a 10T filesystem if possible. So if you have
time to look into the issue I would be willing to recreate the 10T
filesystem and send details.

Regards
Andre
-- 
The only person who always got his work done by Friday was Robinson Crusoe

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: qla2xxx BUG: workqueue leaked lock or atomic

[Valerie Clement]

Mingming Cao wrote:
> On Wed, 2007-03-07 at 11:45 -0800, Andrew Morton wrote:
>> On Wed, 7 Mar 2007 18:09:55 +0100 Andre Noll <maan@systemlinux.org> wrote:
>>
>>> On 20:39, Andrew Morton wrote:
>>>> On Wed, 28 Feb 2007 16:37:22 +0100 Andre Noll <maan@systemlinux.org> wrote:
>>>>
>>> BTW: Are ext3 filesystem sizes greater than 8T now officially
>>> supported?
>> I think so, but I don't know how much 16TB testing developers and
>> distros are doing - perhaps the linux-ext4 denizens can tell us?
>> -
> 
> IBM has done some testing (dbench, fsstress, fsx, tiobench, iozone etc)
> on 10TB ext3, I think RedHat and BULL have done similar test on >8TB
> ext3 too.
> 
> Mingming

Is there not a problem of backward-compatibility with old kernels?
Doesn't we need to handle a new INCOMPAT flag in e2fsprogs and kernel
before allowing ext3 filesystems greater than 8T?

    Valérie

Re: qla2xxx BUG: workqueue leaked lock or atomic

[Andreas Dilger]

On Mar 12, 2007  16:22 +0100, Valerie Clement wrote:
> Mingming Cao wrote:
> >IBM has done some testing (dbench, fsstress, fsx, tiobench, iozone etc)
> >on 10TB ext3, I think RedHat and BULL have done similar test on >8TB
> >ext3 too.
> 
> Is there not a problem of backward-compatibility with old kernels?
> Doesn't we need to handle a new INCOMPAT flag in e2fsprogs and kernel
> before allowing ext3 filesystems greater than 8T?

No, it really depends on the kernel.  There were some bugs that caused
problems with > 8TB because of signed 32-bit int problems, so it isn't
really recommended to use > 8TB unless you know this is fixed in your
kernel (and any older kernel you might have to downgrade to).

Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.

Re: qla2xxx BUG: workqueue leaked lock or atomic

[Valerie Clement]

Andreas Dilger wrote:
> On Mar 12, 2007  16:22 +0100, Valerie Clement wrote:
>> Mingming Cao wrote:
>>> IBM has done some testing (dbench, fsstress, fsx, tiobench, iozone etc)
>>> on 10TB ext3, I think RedHat and BULL have done similar test on >8TB
>>> ext3 too.
>> Is there not a problem of backward-compatibility with old kernels?
>> Doesn't we need to handle a new INCOMPAT flag in e2fsprogs and kernel
>> before allowing ext3 filesystems greater than 8T?
> 
> No, it really depends on the kernel.  There were some bugs that caused
> problems with > 8TB because of signed 32-bit int problems, so it isn't
> really recommended to use > 8TB unless you know this is fixed in your
> kernel (and any older kernel you might have to downgrade to).
> 

OK. Thanks.
As Andre mentions it, it seems that the option "-F" for mkfs is 
necessary to create an ext3 FS > 8T.
(I've got the same behavior but I didn't apply the latest patches 
against my current version of e2fsprogs, so I can't check if that has 
changed since).
Is it the right way?

     Valérie