linux-scsi.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* megaraid_mbox: garbage in file
@ 2006-05-04 18:48 Vasily Averin
  2006-05-04 22:59 ` James Bottomley
  0 siblings, 1 reply; 17+ messages in thread
From: Vasily Averin @ 2006-05-04 18:48 UTC (permalink / raw)
  To: linux-scsi, Neela.Kolli, Atul Mukker, Seokmann.Ju, sreenib,
	James.Bottomley, devel

Hello all,

I've investigated customers claim on the unstable work of their node and found a
strange effect: reading from some files leads to the
 "attempt to access beyond end of device" messages.

I've checked filesystem, memory on the node, motherboard BIOS version, but it
does not help and issue still has been reproduced by simple file reading.

Reproducer is simple:

echo 0xffffffff >/proc/sys/dev/scsi/logging_level ;
cat /vz/private/101/root/etc/ld.so.cache >/tmp/ttt  ;
echo 0 >/proc/sys/dev/scsi/logging

It leads to the following messages in dmesg

sd_init_command: disk=sda, block=871769260, count=26
sda : block=871769260
sda : reading 26/26 512 byte blocks.
scsi_add_timer: scmd: f79ed980, time: 7500, (c02b1420)
sd 0:1:0:0: send 0xf79ed980                  sd 0:1:0:0:
        command: Read (10): 28 00 33 f6 24 ac 00 00 1a 00
buffer = 0xf7cfb540, bufflen = 13312, done = 0xc0366b40, queuecommand 0xc0344010
leaving scsi_dispatch_cmnd()
scsi_delete_timer: scmd: f79ed980, rtn: 1
sd 0:1:0:0: done 0xf79ed980 SUCCESS        0 sd 0:1:0:0:
        command: Read (10): 28 00 33 f6 24 ac 00 00 1a 00
scsi host busy 1 failed 0
sd 0:1:0:0: Notifying upper driver of completion (result 0)
sd_rw_intr: sda: res=0x0
26 sectors total, 13312 bytes done.
use_sg is 4
attempt to access beyond end of device
sda6: rw=0, want=1044134458, limit=951401367
Buffer I/O error on device sda6, logical block 522067228
attempt to access beyond end of device
sda6: rw=0, want=1178878530, limit=951401367
Buffer I/O error on device sda6, logical block 589439264
...

As far as I see first read operation has finished without errors, but when we
read the rest of file we get an access to beyond end of device.

Originally it was found on Virtuozzo kernels (2.6.8.1-based x86 32-bit),
reproduced on RHEL4 kernels 2.6.9-22.EL and 2.6.9-34.EL,
on FC5 (2.6.16-1.2096_FC5) and on vanilla 2.6.16 kernels.

However, when I first read these blocks by using dd with bs=512 or 1024 it works
without any troubles. Then I can cat this file, copy it, map it and so on -- and
get correct content without any errors. Moreover, this issue may be workarounded
by memory limitation: it helps to use mem=4G in kernel commandline or kernels
without PAE support.

I've noticed that we attempt to access to the blocks with a strange numbers:

522067228 = 0x1f1e1d1c
589439264 = 0x23222120 and so on.

Then I've found that I've read strange garbage from file:

# hexdump /tmp/ttt
0000000 0100 0302 0504 0706 0908 0b0a 0d0c 0f0e
0000010 1110 1312 1514 1716 1918 1b1a 1d1c 1f1e
0000020 2120 2322 2524 2726 2928 2b2a 2d2c 2f2e
0000030 3130 3332 3534 3736 3938 3b3a 3d3c 3f3e
0000040 4140 4342 4544 4746 4948 4b4a 4d4c 4f4e
0000050 5150 5352 5554 5756 5958 5b5a 5d5c 5f5e
0000060 6160 6362 6564 6766 6968 6b6a 6d6c 6f6e
0000070 7170 7372 7574 7776 7978 7b7a 7d7c 7f7e
0000080 0100 0302 0504 0706 0908 0b0a 0d0c 0f0e
0000090 1110 1312 1514 1716 1918 1b1a 1d1c 1f1e
00000a0 2120 2322 2524 2726 2928 2b2a 2d2c 2f2e
...
00000f0 7170 7372 7574 7776 7978 7b7a 7d7c 7f7e
0000100 0100 0302 0504 0706 0908 0b0a 0d0c 0f0e
...

Then I've discovered that "access beyond end of device" occurs due reading of
the same garbage from the 13-th (Indirect) block of the file.

I've tried to understand where we got this garbage and found that it is present
in the data buffers beginning at megaraid_mbox driver functions.

Could somebody explain me what is the strange garbage: repeated 0...127?
Seokmann, Atul, could you please tell me if it is a known issue?
James, from my point of view it is not looks like a driver bug, but probably I'm
wrong?

I suppose it is MegaRAID SATA 150-4 firmware issue. I've seen similar firmware
fixes for MegaRAID SATA 300 controllers ("Support PAE mode fixed" and "Fixed the
operating systems using more than 4 gig of memory"). Is it probably the same
issues are present in SATA 150-4 firmware? Or may be I use broken controller?

Hardware Environment:
Tyan B2881
2 x Opteron 246
8G RAM
LSI MegaRAID SATA 150-4
/vz partition formatted as ext3 with 1Kb blocksize

megaraid cmm: 2.20.2.6 (Release Date: Mon Mar 7 00:01:03 EST 2005)
megaraid: 2.20.4.7 (Release Date: Mon Nov 14 12:27:22 EST 2005)
megaraid: probe new device 0x1000:0x1960:0x1000:0x4523: bus 1:slot 4:func 0
ACPI: PCI Interrupt 0000:01:04.0[A] -> GSI 29 (level, low) -> IRQ 16
megaraid: fw version:[713N] bios version:[G119]
scsi0 : LSI Logic MegaRAID driver
scsi[0]: scanning scsi channel 0 [Phy 0] for non-raid devices
scsi[0]: scanning scsi channel 1 [virtual] for logical drives
  Vendor: MegaRAID  Model: LD 0 RAID1  476G  Rev: 713N
  Type:   Direct-Access                      ANSI SCSI revision: 02


Also I would note that from my point of view this issue looks similar to
http://bugzilla.kernel.org/show_bug.cgi?id=6052

It seems for me both of our cases may have the same cause.

Thank you,
	Vasily Averin

SWsoft Virtuozzo/OpenVZ Linux kernel team

^ permalink raw reply	[flat|nested] 17+ messages in thread
* RE: megaraid_mbox: garbage in file
@ 2006-05-05 19:59 Ju, Seokmann
  2006-05-05 23:35 ` Vasily Averin
  0 siblings, 1 reply; 17+ messages in thread
From: Ju, Seokmann @ 2006-05-05 19:59 UTC (permalink / raw)
  To: Vasily Averin, James Bottomley
  Cc: linux-scsi, Kolli, Neela, Mukker, Atul, Bagalkote, Sreenivas,
	devel, Linux Kernel Mailing List

Can you do one quick change in the driver?
Search for 'pci_set_dma_mask()' API calls in the driver and mask out one of them with DMA_64BIT_MASK as follow.
---
	// if (pci_set_dma_mask(adapter->pdev, DMA_64BIT_MASK) != 0) {

	// 	conlog(CL_ANN, (KERN_WARNING
	// 		"megaraid: could not set DMA mask for 64-bit.\n"));

	// 	goto out_free_sysfs_res;
	// }
---

I found that the driver is NOT checking 64-bit DMA capability of the controllers accordingly and this could be a reason.
I'm waiting for feedback from F/W team for MegaRAID 150-4 controller if it supports 64-bit DMA.

I'll update here as I get.

Thank you,

> -----Original Message-----
> From: Vasily Averin [mailto:vvs@sw.ru] 
> Sent: Friday, May 05, 2006 2:17 PM
> To: James Bottomley
> Cc: linux-scsi@vger.kernel.org; Kolli, Neela; Mukker, Atul; 
> Ju, Seokmann; Bagalkote, Sreenivas; devel@openvz.org; Linux 
> Kernel Mailing List
> Subject: Re: megaraid_mbox: garbage in file
> 
> James Bottomley wrote:
> > On Fri, 2006-05-05 at 09:37 +0400, Vasily Averin wrote:
> >>The issue is that the correctly finished scsi read command 
> return me garbage
> >>(repeated 0 ...127 -- see hexdump in my first letter) 
> instead correct file content.
> >>"attempt to access beyond end of device" messages occurs 
> due the same garbage
> >>readed from the Indirect block. I found this garbage 
> present in data buffers
> >>beginning at megaraid driver functions.
> >>
> >>I would note that if I read the same file by using dd with 
> bs=1024 or bs=512 --
> >>I get correct file content.
> >>
> >>When I use kernel with 4Gb memory limit -- the same cat 
> command return me
> >>correct file content too, without any garbage.
> >>
> >>Question is what it is the strange garbage? Have you seen 
> it earlier?
> >>Is it possible that it is some driver-related issue or it 
> is broken hardware?
> >>And why I can workaround this issue by using only 4Gb memory?
> > 
> > This is really odd ... if the controller can't reach *any* 
> memory above
> > 32 bits, then, on an 8GB machine you'd expect corruption 
> all over the
> > place since most user pages come from the top of highmem.
> > 
> > The first thing to try, since you have an opteron system, 
> is to get rid
> > of highmem entirely and use a 64 bit kernel (just to make 
> sure we're not
> > running into some annoying dma_addr_t conversion problem).
> 
> Unfortunately it is customers node, and I'm not able to 
> re-install 64-bit
> distribution to load 64-bit kernel. Of course I'll ask 
> customer about this, but
> it will be done later.
> 
> > Then, I
> > suppose if that doesn't work, try printing out the actual 
> contents of
> > the sg list to see what the physical memory location of the page
> > containing the corrupt block is.
> 
> I've already done such experiment:
> On 2.6.8-based virtuozzo kernel I've added following code to
> megaraid_mbox_display_scb function:
>   virt = page_address(sg[i].page) + sg[i].offset;
>   printk("mbox sg%d: page %p off %d addr %llx len %d "
>          "virt %p first %08x page->flags %08x\n",
>    i, sg[i].page, sg[i].offset, sg[i].dma_address, sg[i].length,
>    virt, virt == NULL ? 0: *(int *)virt, sg[i].page->flags);
> 
> and get the following results
> May  4 02:51:38 vpsn002 kernel:
>  megaraid mailbox: status:0x0 cmd:0xa7 id:0x25 sec:0x1a
> 		 lba:0x33f624ac addr:0xffffffff ld:128 sg:4
>  scsi cmnd: 0x28 0x00 0x33 0xf6 0x24 0xac 0x00 0x00 0x1a 0x00
>  mbox request_buffer eafde340 use_sg 4
>  mbox sg0: page 077a0474 off 0 addr 1fd575000 len 4096 virt ff15a000
> 		 first 03020100 page->flags 40020101
>  mbox sg1: page 077b5738 off 0 addr 1fdede000 len 4096 virt ff141000
> 		 first 03020100 page->flags 40020101
>  mbox sg2: page 077ad500 off 0 addr 1fdb40000 len 4096 virt ff056000
> 		 first 03020100 page->flags 40020101
>  mbox sg3: page 030d46e8 off 1024 addr 5e6a400 len 1024 virt 07e6a400
> 		 first 03020100 page->flags 20001004
> 
> "first 03020100" shows that data in the all sg buffers is 
> already corrupted.
> Also I would note that page for last 1Kb buffer is not Highmem.
> 
> If you want I can reproduce this experiment on 2.6.16 kernel too.
> 
> > This could also be a firmware problem, I suppose, but I 
> haven't seen any
> > similar reports.
> 
> Thank you,
> 	Vasily Averin
> 
> SWsoft Virtuozzo/OpenVZ Linux kernel team
> 

^ permalink raw reply	[flat|nested] 17+ messages in thread
* RE: megaraid_mbox: garbage in file
@ 2006-05-12 12:19 Ju, Seokmann
  0 siblings, 0 replies; 17+ messages in thread
From: Ju, Seokmann @ 2006-05-12 12:19 UTC (permalink / raw)
  To: Vasily Averin
  Cc: James Bottomley, linux-scsi, Kolli, Neela, Mukker, Atul,
	Bagalkote, Sreenivas, devel, Linux Kernel Mailing List

Hi,
Friday, May 12, 2006 12:19 AM, Vasily Averin wrote:
> Could you please tell me any updates? Could you confirm that 
> this issue was
> reproduced on your nodes?
Yes, it has confirmed by F/W team that the controller doesn't support 64-bit DMA.
A patch addresses the issue will followed by soon.

Thank you,

> -----Original Message-----
> From: Vasily Averin [mailto:vvs@sw.ru] 
> Sent: Friday, May 12, 2006 12:19 AM
> To: Vasily Averin
> Cc: Ju, Seokmann; James Bottomley; 
> linux-scsi@vger.kernel.org; Kolli, Neela; Mukker, Atul; 
> Bagalkote, Sreenivas; devel@openvz.org; Linux Kernel Mailing List
> Subject: Re: megaraid_mbox: garbage in file
> 
> Vasily Averin wrote:
> > Ju, Seokmann wrote:
> >>I'm waiting for feedback from F/W team for MegaRAID 150-4 
> controller if it supports 64-bit DMA.
> >>
> >>I'll update here as I get.
> 
> Could you please tell me any updates? Could you confirm that 
> this issue was
> reproduced on your nodes?
> 
> >>Can you do one quick change in the driver?
> >>Search for 'pci_set_dma_mask()' API calls in the driver and 
> mask out one of them with DMA_64BIT_MASK as follow.
> >>---
> >>	// if (pci_set_dma_mask(adapter->pdev, DMA_64BIT_MASK) != 0) {
> >>
> >>	// 	conlog(CL_ANN, (KERN_WARNING
> >>	// 		"megaraid: could not set DMA mask for 
> 64-bit.\n"));
> >>
> >>	// 	goto out_free_sysfs_res;
> >>	// }
> >>---
> >>
> >>I found that the driver is NOT checking 64-bit DMA 
> capability of the controllers accordingly and this could be a reason.
> > 
> > This change help me:
> > Errors go away, file content is correct.
> 
> I'm going to use this change in production, at least as 
> temporal workaround.
> Could you please confirm that it is safe for all controllers 
> supported by this
> driver?
> 
> Thank you,
> 	Vasily Averin
> 
> SWsoft Virtuozzo/OpenVZ Linux kernel team
> 

^ permalink raw reply	[flat|nested] 17+ messages in thread
end of thread, other threads:[~2006-05-16 20:33 UTC | newest]

Thread overview: 17+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2006-05-04 18:48 megaraid_mbox: garbage in file Vasily Averin
2006-05-04 22:59 ` James Bottomley
2006-05-05  5:37   ` Vasily Averin
2006-05-05  9:21     ` Vasily Averin
2006-05-16 17:44       ` [RFC] Megaraid update, submission Andre Hedrick
2006-05-16 18:07         ` Jeff Garzik
2006-05-16 18:13           ` Andre Hedrick
2006-05-16 19:44             ` Matthew Wilcox
2006-05-16 20:24               ` Andre Hedrick
2006-05-05 15:59     ` megaraid_mbox: garbage in file James Bottomley
2006-05-05 18:17       ` Vasily Averin
2006-05-05 20:05         ` James Bottomley
2006-05-05 23:43           ` Vasily Averin
  -- strict thread matches above, loose matches on Subject: below --
2006-05-05 19:59 Ju, Seokmann
2006-05-05 23:35 ` Vasily Averin
2006-05-12  4:18   ` Vasily Averin
2006-05-12 12:19 Ju, Seokmann

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).