Bug in the sg driver

Bug in the sg driver

[Dag Nygren]

Hi,

just got back from a customer with major problems
with a LTO-drive and an Adaptec 19160.

The problems was ones that HP claimed a firmware
upgrade would fix and even gave a tool for doing this:
hp_ltt.

Running this the first time would always segfault and trying
a second time would consistently panic the system.

We traced the segfault to sg_ioctl trying to do something.
After finding a vague hint with google I tried to boot with
mem=512M (The machine has 2GB of memory) and voila:
The update worked without any crashes.
We don't know yet if the firmware update fixed the original problem,
but the conclusion is:

sg_ioctl seems to address illegal parts of memory when used with
kernels where highmem is enabled.

Any sg-driver maintainers out there?

Best Regards

-- 
Dag Nygren                               email: dag@newtech.fi
Oy Espoon NewTech Ab                     phone: +358 9 8024910
Träsktorpet 3                              fax: +358 9 8024916
02360 ESBO                              Mobile: +358 400 426312
FINLAND

Re: Bug in the sg driver

[Randy.Dunlap]

On Mon, 06 Oct 2003 21:06:36 +0300 Dag Nygren <dag@newtech.fi> wrote:

| 
| Hi,
| 
| just got back from a customer with major problems
| with a LTO-drive and an Adaptec 19160.
| 
| The problems was ones that HP claimed a firmware
| upgrade would fix and even gave a tool for doing this:
| hp_ltt.
| 
| Running this the first time would always segfault and trying
| a second time would consistently panic the system.
| 
| We traced the segfault to sg_ioctl trying to do something.
| After finding a vague hint with google I tried to boot with
| mem=512M (The machine has 2GB of memory) and voila:
| The update worked without any crashes.
| We don't know yet if the firmware update fixed the original problem,
| but the conclusion is:
| 
| sg_ioctl seems to address illegal parts of memory when used with
| kernels where highmem is enabled.
| 
| Any sg-driver maintainers out there?


from MAINTAINERS file:
SCSI SG DRIVER
P:	Doug Gilbert
M:	dgilbert@interlog.com
L:	linux-scsi@vger.kernel.org
W:	http://www.torque.net/sg
S:	Maintained


Any details, like kernel version, oops or panic logs, etc.?

--
~Randy

Re: Bug in the sg driver

[Dag Nygren]

> 
> from MAINTAINERS file:
> SCSI SG DRIVER
> P:	Doug Gilbert
> M:	dgilbert@interlog.com
> L:	linux-scsi@vger.kernel.org
> W:	http://www.torque.net/sg
> S:	Maintained

Should I send a bug report to him or is he online here?

> Any details, like kernel version, oops or panic logs, etc.?

Kernel version is 2.4.20 (Redhat 9.0 + newest upgrade)

The oops and the panic logs were not written down as the major focus
was getting this (production) system up and running, sorry for that

 Dag

Re: Bug in the sg driver

[Dag Nygren]

> 
> from MAINTAINERS file:
> SCSI SG DRIVER
> P:	Doug Gilbert
> M:	dgilbert@interlog.com
> L:	linux-scsi@vger.kernel.org
> W:	http://www.torque.net/sg
> S:	Maintained

Should I send a bug report to him or is he online here?

> Any details, like kernel version, oops or panic logs, etc.?

Kernel version is 2.4.20 (Redhat 9.0 + newest upgrade)

The oops and the panic logs were not written down as the major focus
was getting this (production) system up and running, sorry for that

 Dag

Re: Bug in the sg driver

[Randy.Dunlap]

On Mon, 06 Oct 2003 21:47:42 +0300 Dag Nygren <dag@newtech.fi> wrote:

| > 
| > from MAINTAINERS file:
| > SCSI SG DRIVER
| > P:	Doug Gilbert
| > M:	dgilbert@interlog.com
| > L:	linux-scsi@vger.kernel.org
| > W:	http://www.torque.net/sg
| > S:	Maintained
| 
| Should I send a bug report to him or is he online here?

Um, if he's around, he'll see it, although ISTR that he's away
for awhile (like 6 weeks ?).

| > Any details, like kernel version, oops or panic logs, etc.?
| 
| Kernel version is 2.4.20 (Redhat 9.0 + newest upgrade)
| 
| The oops and the panic logs were not written down as the major focus
| was getting this (production) system up and running, sorry for that

Well, unless someone happens to know something about your exact
reported problem....
in general, you will get better help/responses the better your
problem reports are (IMHO).

--
~Randy

Re: Bug in the sg driver

[Randy.Dunlap]

On Mon, 06 Oct 2003 21:47:42 +0300 Dag Nygren <dag@newtech.fi> wrote:

| > 
| > from MAINTAINERS file:
| > SCSI SG DRIVER
| > P:	Doug Gilbert
| > M:	dgilbert@interlog.com
| > L:	linux-scsi@vger.kernel.org
| > W:	http://www.torque.net/sg
| > S:	Maintained
| 
| Should I send a bug report to him or is he online here?

Um, if he's around, he'll see it, although ISTR that he's away
for awhile (like 6 weeks ?).

| > Any details, like kernel version, oops or panic logs, etc.?
| 
| Kernel version is 2.4.20 (Redhat 9.0 + newest upgrade)
| 
| The oops and the panic logs were not written down as the major focus
| was getting this (production) system up and running, sorry for that

Well, unless someone happens to know something about your exact
reported problem....
in general, you will get better help/responses the better your
problem reports are (IMHO).

--
~Randy

Re: Bug in the sg driver

[Dag Nygren]

> | > Any details, like kernel version, oops or panic logs, etc.?
> | 
> | Kernel version is 2.4.20 (Redhat 9.0 + newest upgrade)
> | 
> | The oops and the panic logs were not written down as the major focus
> | was getting this (production) system up and running, sorry for that
> 
> Well, unless someone happens to know something about your exact
> reported problem....
> in general, you will get better help/responses the better your
> problem reports are (IMHO).

Perfectly realize that. Found one report of the exact same problem
on Google. The URL is: 

http://groups.google.fi/groups?q=hp_ltt+linux&hl=sv&lr=&ie=UTF-8&selm=zDJq.3O5.
11%40gated-at.bofh.it&rnum=1

The Oops trace for that is:
kernel BUG in header file at line 162
kernel BUG at panic.c:141!
invalid operand: 0000
CPU:    0
EIP:    0010:[__out_of_line_bug+15/36] Tainted: P
EFLAGS: 00010086
eax: 00000026   ebx: f7928a00   ecx: 00000002 edx: 02000000
esi: c3e8207c   edi: f79ba1f0   ebp: f79ba1d8 esp: f6fabc68
ds: 0018   es: 0018   ss: 0018
Process hp_ltt (pid: 590, stackpage=f6fab000)
Stack: c026fa80 000000a2 c0202f38 000000a2 f7928a00 c3e8207c f79ba1c0 f79ba1d8
       c02c2934 c0132070 00000000 00000000 ffffffff 00000000 0000000e 00000060
       00000000 00000002 c0204831 c3e8207c f7928a00 f79ba1c0 0000005a 00000293
Call Trace:    [mptscsih_AddSGE+200/832] [__alloc_pages+64/352]
 [mptscsih_qcmd+621/1288]  [scsi_dispatch_cmd+649/904] [scsi_old_done+0/1500]
 [scsi_request_fn+826/892]  [__scsi_insert_special+110/128]
 [scsi_insert_special_req+26/32] [scsi_do_req+328/368]
 [sg_common_write+587/604] [sg_cmd_done_bh+0/912] [sg_new_write+539/576]
 [sg_ioctl+616/3004] [journal_dirty_metadata+356/396] [__alloc_pages+64/352]
 [do_wp_page+112/668] [handle_mm_fault+135/184] [do_page_fault+380/1178]
 [do_page_fault+0/1178] [sys_rt_sigaction+159/324] [sys_ioctl+685/746]
 [error_code+52/60] [system_call+51/56]

Code: 0f 0b 8d 00 a6 fa 26 c0 eb fe 8d 74 26 00 8d bc 27 00 00 00

-- Dag

Re: Bug in the sg driver

[Dag Nygren]

> | > Any details, like kernel version, oops or panic logs, etc.?
> | 
> | Kernel version is 2.4.20 (Redhat 9.0 + newest upgrade)
> | 
> | The oops and the panic logs were not written down as the major focus
> | was getting this (production) system up and running, sorry for that
> 
> Well, unless someone happens to know something about your exact
> reported problem....
> in general, you will get better help/responses the better your
> problem reports are (IMHO).

Perfectly realize that. Found one report of the exact same problem
on Google. The URL is: 

http://groups.google.fi/groups?q=hp_ltt+linux&hl=sv&lr=&ie=UTF-8&selm=zDJq.3O5.
11%40gated-at.bofh.it&rnum=1

The Oops trace for that is:
kernel BUG in header file at line 162
kernel BUG at panic.c:141!
invalid operand: 0000
CPU:    0
EIP:    0010:[__out_of_line_bug+15/36] Tainted: P
EFLAGS: 00010086
eax: 00000026   ebx: f7928a00   ecx: 00000002 edx: 02000000
esi: c3e8207c   edi: f79ba1f0   ebp: f79ba1d8 esp: f6fabc68
ds: 0018   es: 0018   ss: 0018
Process hp_ltt (pid: 590, stackpage=f6fab000)
Stack: c026fa80 000000a2 c0202f38 000000a2 f7928a00 c3e8207c f79ba1c0 f79ba1d8
       c02c2934 c0132070 00000000 00000000 ffffffff 00000000 0000000e 00000060
       00000000 00000002 c0204831 c3e8207c f7928a00 f79ba1c0 0000005a 00000293
Call Trace:    [mptscsih_AddSGE+200/832] [__alloc_pages+64/352]
 [mptscsih_qcmd+621/1288]  [scsi_dispatch_cmd+649/904] [scsi_old_done+0/1500]
 [scsi_request_fn+826/892]  [__scsi_insert_special+110/128]
 [scsi_insert_special_req+26/32] [scsi_do_req+328/368]
 [sg_common_write+587/604] [sg_cmd_done_bh+0/912] [sg_new_write+539/576]
 [sg_ioctl+616/3004] [journal_dirty_metadata+356/396] [__alloc_pages+64/352]
 [do_wp_page+112/668] [handle_mm_fault+135/184] [do_page_fault+380/1178]
 [do_page_fault+0/1178] [sys_rt_sigaction+159/324] [sys_ioctl+685/746]
 [error_code+52/60] [system_call+51/56]

Code: 0f 0b 8d 00 a6 fa 26 c0 eb fe 8d 74 26 00 8d bc 27 00 00 00

-- Dag

Re: Bug in the sg driver

[Randy.Dunlap]

On Mon, 06 Oct 2003 22:11:35 +0300 Dag Nygren <dag@newtech.fi> wrote:

| 
| > | > Any details, like kernel version, oops or panic logs, etc.?
| > | 
| > | Kernel version is 2.4.20 (Redhat 9.0 + newest upgrade)
| > | 
| > | The oops and the panic logs were not written down as the major focus
| > | was getting this (production) system up and running, sorry for that
| > 
| > Well, unless someone happens to know something about your exact
| > reported problem....
| > in general, you will get better help/responses the better your
| > problem reports are (IMHO).
| 
| Perfectly realize that. Found one report of the exact same problem
| on Google. The URL is: 
| 
| http://groups.google.fi/groups?q=hp_ltt+linux&hl=sv&lr=&ie=UTF-8&selm=zDJq.3O5.
| 11%40gated-at.bofh.it&rnum=1
| 
| The Oops trace for that is:
| kernel BUG in header file at line 162
| kernel BUG at panic.c:141!
| invalid operand: 0000
| CPU:    0
| EIP:    0010:[__out_of_line_bug+15/36] Tainted: P
| EFLAGS: 00010086
| eax: 00000026   ebx: f7928a00   ecx: 00000002 edx: 02000000
| esi: c3e8207c   edi: f79ba1f0   ebp: f79ba1d8 esp: f6fabc68
| ds: 0018   es: 0018   ss: 0018
| Process hp_ltt (pid: 590, stackpage=f6fab000)
| Stack: c026fa80 000000a2 c0202f38 000000a2 f7928a00 c3e8207c f79ba1c0 f79ba1d8
|        c02c2934 c0132070 00000000 00000000 ffffffff 00000000 0000000e 00000060
|        00000000 00000002 c0204831 c3e8207c f7928a00 f79ba1c0 0000005a 00000293
| Call Trace:    [mptscsih_AddSGE+200/832] [__alloc_pages+64/352]
|  [mptscsih_qcmd+621/1288]  [scsi_dispatch_cmd+649/904] [scsi_old_done+0/1500]
|  [scsi_request_fn+826/892]  [__scsi_insert_special+110/128]
|  [scsi_insert_special_req+26/32] [scsi_do_req+328/368]
|  [sg_common_write+587/604] [sg_cmd_done_bh+0/912] [sg_new_write+539/576]
|  [sg_ioctl+616/3004] [journal_dirty_metadata+356/396] [__alloc_pages+64/352]
|  [do_wp_page+112/668] [handle_mm_fault+135/184] [do_page_fault+380/1178]
|  [do_page_fault+0/1178] [sys_rt_sigaction+159/324] [sys_ioctl+685/746]
|  [error_code+52/60] [system_call+51/56]
| 
| Code: 0f 0b 8d 00 a6 fa 26 c0 eb fe 8d 74 26 00 8d bc 27 00 00 00

Is your oops/panic like this one?

mptscsih_qcmd is in the Fusion MPT driver.
Looks like it is having problems with a scatter-gather list
(actually with PCI mapping of it).
I can't tell if the Fusion driver begins the problem or if
sg begins it and passes it along.

Can you try a current 2.4.22 kernel?  This driver has been updated
some since 2.4.20.  I don't know that this bug is fixed, however.

If you get this bug again, please run the logged text thru ksymoops
so that it is more usable.

What makes the kernel "Tainted"?

--
~Randy

Re: Bug in the sg driver

[Dag Nygren]

> On Mon, 06 Oct 2003 22:11:35 +0300 Dag Nygren <dag@newtech.fi> wrote:
> 
> 
> Is your oops/panic like this one?

Yep, at least from sg_ioctl down.

> mptscsih_qcmd is in the Fusion MPT driver.
> Looks like it is having problems with a scatter-gather list
> (actually with PCI mapping of it).
> I can't tell if the Fusion driver begins the problem or if
> sg begins it and passes it along.

Our driver is the aic7xxx, the board is an Adaptec 19160.

> Can you try a current 2.4.22 kernel?  This driver has been updated
> some since 2.4.20.  I don't know that this bug is fixed, however.

Not until Redhat compiles one, as we don't want to use anything
not Redhat for this installation.

> If you get this bug again, please run the logged text thru ksymoops
> so that it is more usable.

Yes, and we will also try to capture "our" oops this time.
Coming to think about it, it is probably logged somewhere...
Not the successive Panic, but the oops should be there. I will investigate
a bit..

> What makes the kernel "Tainted"?

Our kernel is a stock Redhat 9.0 kernel and shouldn't be tainted.

-- Dag

Re: Bug in the sg driver

[Matthew Wilcox]

On Tue, Oct 07, 2003 at 08:47:00AM +0300, Dag Nygren wrote:
> > Can you try a current 2.4.22 kernel?  This driver has been updated
> > some since 2.4.20.  I don't know that this bug is fixed, however.
> 
> Not until Redhat compiles one, as we don't want to use anything
> not Redhat for this installation.

Then you really should be taking these problems up with Red Hat,
not with us.

-- 
"It's not Hollywood.  War is real, war is primarily not about defeat or
victory, it is about death.  I've seen thousands and thousands of dead bodies.
Do you think I want to have an academic debate on this subject?" -- Robert Fisk

Re: Bug in the sg driver

[Dag Nygren]

> On Tue, Oct 07, 2003 at 08:47:00AM +0300, Dag Nygren wrote:
 
> Then you really should be taking these problems up with Red Hat,
> not with us.

So you mean that this bug is squished in 2.4.22 then? 

Anyway I got an oops-trace ran through ksymoops by the
customer and here is the result (If anyone is interested anymore....)

ksymoops 2.4.9 on i686 2.4.20-20.9.  Options used
     -V (default)
     -k /proc/ksyms (default)
     -l /proc/modules (default)
     -o /lib/modules/2.4.20-20.9/ (default)
     -m /usr/src/linux/System.map (default)

Warning: You did not tell me where to find symbol information.  I will
assume that the log matches the kernel and modules that are running
right now and I'll use the default options above for symbol resolution.
If the current kernel and/or modules do not match the log, you can get
more accurate output by telling me the kernel version and where to find
map, modules, ksyms etc.  ksymoops -h explains the options.

Error (expand_objects): cannot stat(/lib/ext3.o) for ext3
Error (expand_objects): cannot stat(/lib/jbd.o) for jbd
Error (expand_objects): cannot stat(/lib/aacraid.o) for aacraid
Error (expand_objects): cannot stat(/lib/aic7xxx.o) for aic7xxx
Error (expand_objects): cannot stat(/lib/sd_mod.o) for sd_mod
Error (expand_objects): cannot stat(/lib/scsi_mod.o) for scsi_mod
Error (regular_file): read_system_map stat /usr/src/linux/System.map failed
Warning (map_ksym_to_module): cannot match loaded module ext3 to a unique 
module object.  Trace may not be reliable.
Warning (map_ksym_to_module): cannot match loaded module aacraid to a unique 
module object.  Trace may not be reliable.
Oct  6 19:37:37 venus kernel: kernel BUG at panic.c:288!
Oct  6 19:37:37 venus kernel: invalid operand: 0000
Oct  6 19:37:37 venus kernel: CPU:    0
Oct  6 19:37:37 venus kernel: EIP:    0060:[<c011bec7>]    Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
Oct  6 19:37:37 venus kernel: EFLAGS: 00010082
Oct  6 19:37:37 venus kernel: EIP is at __out_of_line_bug [kernel] 0x17 
(2.4.20-20.9)
Oct  6 19:37:37 venus kernel: eax: 00000026   ebx: 00000014   ecx: c037459c   
edx: 00000046
Oct  6 19:37:37 venus kernel: esi: f55b8000   edi: 00000011   ebp: c47a8068   
esp: f5cc3c44
Oct  6 19:37:37 venus kernel: ds: 0068   es: 0068   ss: 0068
Oct  6 19:37:37 venus kernel: Process hp_ltt (pid: 3813, stackpage=f5cc3000)
Oct  6 19:37:37 venus kernel: Stack: c0260b80 000000a2 f882ff34 000000a2 
f55b8000 00000246 c46c42b0 00000020
Oct  6 19:37:37 venus kernel:        416c4f58 c47a4080 00000001 c4659c00 
00000000 00000001 f6f2f600 f882f71d
Oct  6 19:37:37 venus kernel:        c4659c00 c4604380 00000000 00000000 
00000086 00000297 c46e32d4 f6f2f600
Oct  6 19:37:37 venus kernel: Call Trace:   [<f882ff34>] 
ahc_linux_run_device_queue [aic7xxx] 0x784 (0xf5cc3c4c))
Oct  6 19:37:37 venus kernel: [<f882f71d>] ahc_linux_queue [aic7xxx] 0x14d 
(0xf5cc3c80))
Oct  6 19:37:37 venus kernel: [<f880d6c2>] scsi_dispatch_cmd [scsi_mod] 0x112 
(0xf5cc3ca8))
Oct  6 19:37:37 venus kernel: [<f880dfc0>] scsi_done [scsi_mod] 0x0 
(0xf5cc3cb0))
Oct  6 19:37:37 venus kernel: [<f8812820>] scsi_times_out [scsi_mod] 0x0 
(0xf5cc3cb4))
Oct  6 19:37:37 venus kernel: [<f8816446>] scsi_request_fn [scsi_mod] 0x1d6 
(0xf5cc3ce0))
Oct  6 19:37:37 venus kernel: [<f8815818>] __scsi_insert_special [scsi_mod] 
0x58 (0xf5cc3d18))
Oct  6 19:37:38 venus kernel: [<f8815898>] scsi_insert_special_req [scsi_mod] 
0x28 (0xf5cc3d28))
Oct  6 19:37:38 venus kernel: [<f880daab>] scsi_do_req_R1f341175 [scsi_mod] 
0xeb (0xf5cc3d3c))
Oct  6 19:37:38 venus kernel: [<f896f320>] sg_cmd_done_bh [sg] 0x0 
(0xf5cc3d70))
Oct  6 19:37:38 venus kernel: [<f896e134>] sg_common_write [sg] 0x1f4 
(0xf5cc3d8c))
Oct  6 19:37:38 venus kernel: [<f896f320>] sg_cmd_done_bh [sg] 0x0 
(0xf5cc3da0))
Oct  6 19:37:38 venus kernel: [<f896de7b>] sg_new_write [sg] 0x1eb 
(0xf5cc3dbc))
Oct  6 19:37:38 venus kernel: [<f896ebe8>] sg_ioctl [sg] 0xa28 (0xf5cc3e00))
Oct  6 19:37:38 venus kernel: [<c0145378>] kmap_high [kernel] 0x48 
(0xf5cc3e50))
Oct  6 19:37:38 venus kernel: [<c013f01d>] __alloc_pages [kernel] 0x7d 
(0xf5cc3e70))
Oct  6 19:37:38 venus kernel: [<c012eceb>] vm_set_pte [kernel] 0x3b 
(0xf5cc3e90))
Oct  6 19:37:38 venus kernel: [<c0130227>] do_wp_page [kernel] 0x337 
(0xf5cc3eb4))
Oct  6 19:37:38 venus kernel: [<c0130ce0>] handle_mm_fault [kernel] 0x120 
(0xf5cc3ed8))
Oct  6 19:37:38 venus kernel: [<c011750c>] do_page_fault [kernel] 0x16c 
(0xf5cc3f08))
Oct  6 19:37:38 venus kernel: [<c012884b>] sys_rt_sigaction [kernel] 0x8b 
(0xf5cc3f60))
Oct  6 19:37:38 venus kernel: [<c012016e>] sys_wait4 [kernel] 0x1ce 
(0xf5cc3f74))
Oct  6 19:37:38 venus kernel: [<c0156ff9>] sys_ioctl [kernel] 0xc9 
(0xf5cc3f94))
Oct  6 19:37:38 venus kernel: [<c010953f>] system_call [kernel] 0x33 
(0xf5cc3fc0))
Oct  6 19:37:38 venus kernel: Code: 0f 0b 20 01 03 04 26 c0 90 eb fe 90 90 90 
90 90 90 90 90 90


>>EIP; c011bec7 <__out_of_line_bug+17/600>   <=====

>>ecx; c037459c <init_task_union+2459c/24a80>
>>esi; f55b8000 <___strtok+351d8e84/3842dee4>
>>ebp; c47a8068 <___strtok+43c8eec/3842dee4>
>>esp; f5cc3c44 <___strtok+358e4ac8/3842dee4>

Trace; f882ff34 <[aic7xxx]ahc_linux_run_device_queue+784/900>
Trace; f882f71d <[aic7xxx]ahc_linux_queue+14d/1e0>
Trace; f880d6c2 <[scsi_mod]scsi_dispatch_cmd+112/360>
Trace; f880dfc0 <[scsi_mod]scsi_done+0/d0>
Trace; f8812820 <[scsi_mod]scsi_times_out+0/d0>
Trace; f8816446 <[scsi_mod]scsi_request_fn+1d6/3a0>
Trace; f8815818 <[scsi_mod]__scsi_insert_special+58/80>
Trace; f8815898 <[scsi_mod]scsi_insert_special_req+28/30>
Trace; f880daab <[scsi_mod]scsi_do_req+eb/1e0>
Trace; f896f320 <[sg]sg_cmd_done_bh+0/360>
Trace; f896e134 <[sg]sg_common_write+1f4/280>
Trace; f896f320 <[sg]sg_cmd_done_bh+0/360>
Trace; f896de7b <[sg]sg_new_write+1eb/2b0>
Trace; f896ebe8 <[sg]sg_ioctl+a28/c00>
Trace; c0145378 <kmap_high+48/50>
Trace; c013f01d <__alloc_pages+7d/370>
Trace; c012eceb <cpufreq_frequency_table_setpolicy+1fb/ac0>
Trace; c0130227 <remap_page_range+587/630>
Trace; c0130ce0 <vmtruncate+a10/b00>
Trace; c011750c <__verify_write+30c/8e0>
Trace; c012884b <notify_parent+115b/1a60>
Trace; c012016e <sys_wait4+1ce/1870>
Trace; c0156ff9 <kill_fasync+269/470>
Trace; c010953f <__up_wakeup+109f/1480>

Code;  c011bec7 <__out_of_line_bug+17/600>
00000000 <_EIP>:
Code;  c011bec7 <__out_of_line_bug+17/600>   <=====
   0:   0f 0b                     ud2a      <=====
Code;  c011bec9 <__out_of_line_bug+19/600>
   2:   20 01                     and    %al,(%ecx)
Code;  c011becb <__out_of_line_bug+1b/600>
   4:   03 04 26                  add    (%esi,1),%eax
Code;  c011bece <__out_of_line_bug+1e/600>
   7:   c0 90 eb fe 90 90 90      rclb   $0x90,0x9090feeb(%eax)
Code;  c011bed5 <__out_of_line_bug+25/600>
   e:   90                        nop    
Code;  c011bed6 <__out_of_line_bug+26/600>
   f:   90                        nop    
Code;  c011bed7 <__out_of_line_bug+27/600>
  10:   90                        nop    
Code;  c011bed8 <__out_of_line_bug+28/600>
  11:   90                        nop    
Code;  c011bed9 <__out_of_line_bug+29/600>
  12:   90                        nop    
Code;  c011beda <__out_of_line_bug+2a/600>
  13:   90                        nop    


3 warnings and 7 errors issued.  Results may not be reliable.


Best for now.

-- Dag

Re: Bug in the sg driver

[Matthew Wilcox]

On Tue, Oct 07, 2003 at 03:03:55PM +0300, Dag Nygren wrote:
> > On Tue, Oct 07, 2003 at 08:47:00AM +0300, Dag Nygren wrote:
>  
> > Then you really should be taking these problems up with Red Hat,
> > not with us.
> 
> So you mean that this bug is squished in 2.4.22 then? 

Not necessarily.  Red Hat's kernels contain a lot of customisations and
changes versus the stock Linux tree.  It's almost impossible for us to
debug problems in Red Hat's kernel.  It's possible (though unlikely)
the bug was even introduced by Red Hat.

In any case, if you need a Red Hat supplied kernel, you need to work
with Red Hat.  There's really no way around that.

-- 
"It's not Hollywood.  War is real, war is primarily not about defeat or
victory, it is about death.  I've seen thousands and thousands of dead bodies.
Do you think I want to have an academic debate on this subject?" -- Robert Fisk

Re: Bug in the sg driver

[Douglas Gilbert]

Dag Nygren wrote:
>>On Tue, Oct 07, 2003 at 08:47:00AM +0300, Dag Nygren wrote:
> 
>  
> 
>>Then you really should be taking these problems up with Red Hat,
>>not with us.
> 
> 
> So you mean that this bug is squished in 2.4.22 then? 
> 
> Anyway I got an oops-trace ran through ksymoops by the
> customer and here is the result (If anyone is interested anymore....)
> 
> ksymoops 2.4.9 on i686 2.4.20-20.9.  Options used
>      -V (default)
>      -k /proc/ksyms (default)
>      -l /proc/modules (default)
>      -o /lib/modules/2.4.20-20.9/ (default)
>      -m /usr/src/linux/System.map (default)
> 
> Warning: You did not tell me where to find symbol information.  I will
> assume that the log matches the kernel and modules that are running
> right now and I'll use the default options above for symbol resolution.
> If the current kernel and/or modules do not match the log, you can get
> more accurate output by telling me the kernel version and where to find
> map, modules, ksyms etc.  ksymoops -h explains the options.
> 
> Error (expand_objects): cannot stat(/lib/ext3.o) for ext3
> Error (expand_objects): cannot stat(/lib/jbd.o) for jbd
> Error (expand_objects): cannot stat(/lib/aacraid.o) for aacraid
> Error (expand_objects): cannot stat(/lib/aic7xxx.o) for aic7xxx
> Error (expand_objects): cannot stat(/lib/sd_mod.o) for sd_mod
> Error (expand_objects): cannot stat(/lib/scsi_mod.o) for scsi_mod
> Error (regular_file): read_system_map stat /usr/src/linux/System.map failed
> Warning (map_ksym_to_module): cannot match loaded module ext3 to a unique 
> module object.  Trace may not be reliable.
> Warning (map_ksym_to_module): cannot match loaded module aacraid to a unique 
> module object.  Trace may not be reliable.
> Oct  6 19:37:37 venus kernel: kernel BUG at panic.c:288!
> Oct  6 19:37:37 venus kernel: invalid operand: 0000
> Oct  6 19:37:37 venus kernel: CPU:    0
> Oct  6 19:37:37 venus kernel: EIP:    0060:[<c011bec7>]    Not tainted
> Using defaults from ksymoops -t elf32-i386 -a i386
> Oct  6 19:37:37 venus kernel: EFLAGS: 00010082
> Oct  6 19:37:37 venus kernel: EIP is at __out_of_line_bug [kernel] 0x17 
> (2.4.20-20.9)
> Oct  6 19:37:37 venus kernel: eax: 00000026   ebx: 00000014   ecx: c037459c   
> edx: 00000046
> Oct  6 19:37:37 venus kernel: esi: f55b8000   edi: 00000011   ebp: c47a8068   
> esp: f5cc3c44
> Oct  6 19:37:37 venus kernel: ds: 0068   es: 0068   ss: 0068
> Oct  6 19:37:37 venus kernel: Process hp_ltt (pid: 3813, stackpage=f5cc3000)
> Oct  6 19:37:37 venus kernel: Stack: c0260b80 000000a2 f882ff34 000000a2 
> f55b8000 00000246 c46c42b0 00000020
> Oct  6 19:37:37 venus kernel:        416c4f58 c47a4080 00000001 c4659c00 
> 00000000 00000001 f6f2f600 f882f71d
> Oct  6 19:37:37 venus kernel:        c4659c00 c4604380 00000000 00000000 
> 00000086 00000297 c46e32d4 f6f2f600
> Oct  6 19:37:37 venus kernel: Call Trace:   [<f882ff34>] 
> ahc_linux_run_device_queue [aic7xxx] 0x784 (0xf5cc3c4c))
> Oct  6 19:37:37 venus kernel: [<f882f71d>] ahc_linux_queue [aic7xxx] 0x14d 
> (0xf5cc3c80))
> Oct  6 19:37:37 venus kernel: [<f880d6c2>] scsi_dispatch_cmd [scsi_mod] 0x112 
> (0xf5cc3ca8))
> Oct  6 19:37:37 venus kernel: [<f880dfc0>] scsi_done [scsi_mod] 0x0 
> (0xf5cc3cb0))
> Oct  6 19:37:37 venus kernel: [<f8812820>] scsi_times_out [scsi_mod] 0x0 
> (0xf5cc3cb4))
> Oct  6 19:37:37 venus kernel: [<f8816446>] scsi_request_fn [scsi_mod] 0x1d6 
> (0xf5cc3ce0))
> Oct  6 19:37:37 venus kernel: [<f8815818>] __scsi_insert_special [scsi_mod] 
> 0x58 (0xf5cc3d18))
> Oct  6 19:37:38 venus kernel: [<f8815898>] scsi_insert_special_req [scsi_mod] 
> 0x28 (0xf5cc3d28))
> Oct  6 19:37:38 venus kernel: [<f880daab>] scsi_do_req_R1f341175 [scsi_mod] 
> 0xeb (0xf5cc3d3c))
> Oct  6 19:37:38 venus kernel: [<f896f320>] sg_cmd_done_bh [sg] 0x0 
> (0xf5cc3d70))
> Oct  6 19:37:38 venus kernel: [<f896e134>] sg_common_write [sg] 0x1f4 
> (0xf5cc3d8c))
> Oct  6 19:37:38 venus kernel: [<f896f320>] sg_cmd_done_bh [sg] 0x0 
> (0xf5cc3da0))
> Oct  6 19:37:38 venus kernel: [<f896de7b>] sg_new_write [sg] 0x1eb 
> (0xf5cc3dbc))
> Oct  6 19:37:38 venus kernel: [<f896ebe8>] sg_ioctl [sg] 0xa28 (0xf5cc3e00))
> Oct  6 19:37:38 venus kernel: [<c0145378>] kmap_high [kernel] 0x48 
> (0xf5cc3e50))
> Oct  6 19:37:38 venus kernel: [<c013f01d>] __alloc_pages [kernel] 0x7d 
> (0xf5cc3e70))
> Oct  6 19:37:38 venus kernel: [<c012eceb>] vm_set_pte [kernel] 0x3b 
> (0xf5cc3e90))
> Oct  6 19:37:38 venus kernel: [<c0130227>] do_wp_page [kernel] 0x337 
> (0xf5cc3eb4))
> Oct  6 19:37:38 venus kernel: [<c0130ce0>] handle_mm_fault [kernel] 0x120 
> (0xf5cc3ed8))
> Oct  6 19:37:38 venus kernel: [<c011750c>] do_page_fault [kernel] 0x16c 
> (0xf5cc3f08))
> Oct  6 19:37:38 venus kernel: [<c012884b>] sys_rt_sigaction [kernel] 0x8b 
> (0xf5cc3f60))
> Oct  6 19:37:38 venus kernel: [<c012016e>] sys_wait4 [kernel] 0x1ce 
> (0xf5cc3f74))
> Oct  6 19:37:38 venus kernel: [<c0156ff9>] sys_ioctl [kernel] 0xc9 
> (0xf5cc3f94))
> Oct  6 19:37:38 venus kernel: [<c010953f>] system_call [kernel] 0x33 
> (0xf5cc3fc0))
> Oct  6 19:37:38 venus kernel: Code: 0f 0b 20 01 03 04 26 c0 90 eb fe 90 90 90 
> 90 90 90 90 90 90
> 
> 
> 
>>>EIP; c011bec7 <__out_of_line_bug+17/600>   <=====
> 
> 
>>>ecx; c037459c <init_task_union+2459c/24a80>
>>>esi; f55b8000 <___strtok+351d8e84/3842dee4>
>>>ebp; c47a8068 <___strtok+43c8eec/3842dee4>
>>>esp; f5cc3c44 <___strtok+358e4ac8/3842dee4>
> 
> 
> Trace; f882ff34 <[aic7xxx]ahc_linux_run_device_queue+784/900>
> Trace; f882f71d <[aic7xxx]ahc_linux_queue+14d/1e0>
> Trace; f880d6c2 <[scsi_mod]scsi_dispatch_cmd+112/360>
> Trace; f880dfc0 <[scsi_mod]scsi_done+0/d0>
> Trace; f8812820 <[scsi_mod]scsi_times_out+0/d0>
> Trace; f8816446 <[scsi_mod]scsi_request_fn+1d6/3a0>
> Trace; f8815818 <[scsi_mod]__scsi_insert_special+58/80>
> Trace; f8815898 <[scsi_mod]scsi_insert_special_req+28/30>
> Trace; f880daab <[scsi_mod]scsi_do_req+eb/1e0>
> Trace; f896f320 <[sg]sg_cmd_done_bh+0/360>
> Trace; f896e134 <[sg]sg_common_write+1f4/280>
> Trace; f896f320 <[sg]sg_cmd_done_bh+0/360>
> Trace; f896de7b <[sg]sg_new_write+1eb/2b0>
> Trace; f896ebe8 <[sg]sg_ioctl+a28/c00>
> Trace; c0145378 <kmap_high+48/50>
> Trace; c013f01d <__alloc_pages+7d/370>
> Trace; c012eceb <cpufreq_frequency_table_setpolicy+1fb/ac0>
> Trace; c0130227 <remap_page_range+587/630>
> Trace; c0130ce0 <vmtruncate+a10/b00>
> Trace; c011750c <__verify_write+30c/8e0>
> Trace; c012884b <notify_parent+115b/1a60>
> Trace; c012016e <sys_wait4+1ce/1870>
> Trace; c0156ff9 <kill_fasync+269/470>
> Trace; c010953f <__up_wakeup+109f/1480>
> 
> Code;  c011bec7 <__out_of_line_bug+17/600>
> 00000000 <_EIP>:
> Code;  c011bec7 <__out_of_line_bug+17/600>   <=====
>    0:   0f 0b                     ud2a      <=====
> Code;  c011bec9 <__out_of_line_bug+19/600>
>    2:   20 01                     and    %al,(%ecx)
> Code;  c011becb <__out_of_line_bug+1b/600>
>    4:   03 04 26                  add    (%esi,1),%eax
> Code;  c011bece <__out_of_line_bug+1e/600>
>    7:   c0 90 eb fe 90 90 90      rclb   $0x90,0x9090feeb(%eax)
> Code;  c011bed5 <__out_of_line_bug+25/600>
>    e:   90                        nop    
> Code;  c011bed6 <__out_of_line_bug+26/600>
>    f:   90                        nop    
> Code;  c011bed7 <__out_of_line_bug+27/600>
>   10:   90                        nop    
> Code;  c011bed8 <__out_of_line_bug+28/600>
>   11:   90                        nop    
> Code;  c011bed9 <__out_of_line_bug+29/600>
>   12:   90                        nop    
> Code;  c011beda <__out_of_line_bug+2a/600>
>   13:   90                        nop    
> 
> 
> 3 warnings and 7 errors issued.  Results may not be reliable.

Dag,
I presume the hp_ltt program is controlling a tape
robot. hp_ltt sent a command to sg which passed it
through the mid level to the aic7xxx driver which
crashed. Hard to say why.

New aic7xxx drivers can be found at:
    http://people.freebsd.org/~gibbs/linux
At the bottom of that page are newer rpms for RedHat 9
(than distributed by redhat).

Doug Gilbert

Re: Bug in the sg driver

[Dag Nygren]

Thanks for this response, this is of the kind that could
be useful.

> Dag,
> I presume the hp_ltt program is controlling a tape
> robot. hp_ltt sent a command to sg which passed it
> through the mid level to the aic7xxx driver which
> crashed. Hard to say why.

hp_ltt is a utility from HP to control their tapedrives,
ie. update firmware, diagnostic functions and so on.

The strange thing is that the URL I gave earlier in this thread 
would indicate the same behaviour with a completely
different SCSI-driver....

> New aic7xxx drivers can be found at:
>     http://people.freebsd.org/~gibbs/linux
> At the bottom of that page are newer rpms for RedHat 9
> (than distributed by redhat).

Thanks for that, I could try this out at some point.
Now the update of the tapedrive could be done by adding mem=512M
to the boot line, running the firmware update and then reboot again wo.
the memory limit.
So this was meant more like a report that something is wrong in there.

--- Dag